Are there any plans for a PushShift event sourtce?

Help
, , ,
1

This topic was automatically generated from Slack. You can find the original thread here.

Adam Gordon Bell : Is there a way to use the reddit / pipeshift source like rss or twitter search, so that each result is a event and only new ones trigger?

2

Dylan Sather (Pipedream) : which specific Reddit source are you using? Are you not seeing individual events emitted (i.e. are you seeing events emitted in batch?), or are you seeing duplicate events?

3

Adam Gordon Bell : I guess its not a source, its Search Reddit posts using the [Pushshift.io](http://Pushshift.io) API. Learn more at https://github.com/pushshift/api

4

Dylan Sather (Pipedream) : Ah gotcha. We do operate some Reddit triggers / sources for new posts, comments, etc: Reddit API Integrations - Pipedream . These should trigger the workflow when a new relevant event is seen, and should only emit new events. Would any of those sources solve your use case?

5

Adam Gordon Bell : I didn’t see search, so I did try getting an RSS feed for a search feed out of reddit and used the RSS source, but Reddits search has some issues.

6

Adam Gordon Bell : So a search turned up the pushshift reddit thing, which seems perfect, just it doesn’t really come out the other side as chopped up events.

7

Adam Gordon Bell : I’m sure this is where you say you are looking for contributors :slightly_smiling_face:

8

Adam Gordon Bell : The search problem is this, I’m looking for mentions of my company, earthly, but reddit search uses stemming, so you can only search for ‘earth’ which is not really a solution.

9

Dylan Sather (Pipedream) : I see. Yes, Pushshift is a great solution and we would love to see someone contribute a Pushshift source!

10

Dylan Sather (Pipedream) : Do you mind opening a feature request for now that captures specifically what you’d like to see? Issues · PipedreamHQ/pipedream · GitHub

11

Dylan Sather (Pipedream) : I’m happy to help you author the source if you’d like, otherwise we can add it to our backlog

12

Adam Gordon Bell : I put the issue in here: [FEATURE] PushShift Source · Issue #1406 · PipedreamHQ/pipedream · GitHub

13

Adam Gordon Bell : I’d love to help contribute to features I need, but its unclear to me when I’ll have the time to dig in on attempting to do so.