Tuesday, February 14, 2012

Best way to handle existing rows

I need to create a fairly simple package that copies data from one database using a sql querie and inserts it into another. what's the best way to handle rows that already exist in the destination table? The package needs to run once a day and ignore rows inserted from previous runs.You can pass the data through a lookup transform and compare them to the destination dataset. The rows that don't match will be marked as errors so click on the "configure the error output" and set it to redirect rows instead of failure. Then you can connect the red arrow to your destination and map the rows. There will be two extra columns but you don't need to worry about them as they are just for error tracking and you don't technically have errors.|||

The method brent describes is discussed in more detail here: http://www.sqlis.com/default.aspx?311

Its also worth saying that if the data in the source is timestamped when it canges then you can use that timestamp to only pull out data changed since previous load. i.e. Reduce the amount of data that you have to process.

-Jamie

No comments:

Post a Comment