Geared Spacy: Building NLP pipeline in RedisGears

Initially, I only planned to use Redis Cluster as one large in-memory store/database, store all article ids in the set corresponding to the pipeline step and then run: for each article_id in a set, apply next step in the pipeline, save id into next set. Potentially explore nomad to distribute compute.

Next few steps are memory demanding and I mean it, but the temptation to use RedisGears to distribute calculation over Redis Cluster with no effort on my side was too great. I even didn’t finish watching Tutorial, when I started coding (hint: watch…