Data Maintance

[kordianbruck]

Hey guys,

one thing that might still be open: can the scraping cli command be run multiple times without inserting doubles?

We want to maintain the data in the coming 2-3 years at least. So that running the CLI multiple times should ensure that it checks if the rows are already in the database and update the entity accordingly.

We want to put this into a cronjob and run each week or month.

Thanks

[sacdallago]

http://docs.sequelizejs.com/en/latest/api/model/#upsertvalues-options-promisecreated might be a place to start, substituting inserts, but only if you have a logic in place to match two documents. You might eventually need to create a distance between to objects dictated by the object's fields (or features) and define a threshold by which you consider two objects similar :) but, but. For now, I would say that upsert instead of create should do, based on name & birthdate matching?