Skip to content

Insert only if Unique. #332

Closed
Closed

Description

@DogWithWings

Hi all,

I am using ODBC.load to push updated data into a database. This unfortunately fails if the dataframe contains elements already present in the database.

It would be great is ODBC.load had a "skip existing" option which ignored rows for which the primary key already exists. This would likely require passing the unique column names to the function as well.

Example of the error:

ERROR: LoadError: 40002: [MonetDB][ODBC Driver 11.43.5]INSERT INTO: PRIMARY KEY constraint 'weather_history.weather_history_station_id_date_pkey' violated
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] execute(stmt::ODBC.API.Handle)
   @ ODBC.API C:\Users\rjeffrey003\.julia\packages\ODBC\qhwMX\src\API.jl:114
 [3] execute(stmt::ODBC.Statement, params::Tables.Row{DataFrameRow{DataFrame, DataFrames.Index}}; debug::Bool, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ ODBC C:\Users\rjeffrey003\.julia\packages\ODBC\qhwMX\src\dbinterface.jl:164
 [4] (::ODBC.var"#46#47"{Bool, Int64, ODBC.Connection})()
   @ ODBC C:\Users\rjeffrey003\.julia\packages\ODBC\qhwMX\src\load.jl:102
 [5] transaction(f::ODBC.var"#46#47"{Bool, Int64, ODBC.Connection}, conn::ODBC.Connection)
   @ ODBC C:\Users\rjeffrey003\.julia\packages\ODBC\qhwMX\src\load.jl:112
 [6] load(itr::DataFrame, conn::ODBC.Connection, name::String; append::Bool, quoteidentifiers::Bool, debug::Bool, limit::Int64, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ ODBC C:\Users\rjeffrey003\.julia\packages\ODBC\qhwMX\src\load.jl:96
 [7] load(itr::DataFrame, conn::ODBC.Connection, name::String)
   @ ODBC C:\Users\rjeffrey003\.julia\packages\ODBC\qhwMX\src\load.jl:76
 [8] top-level scope
   @ c:\Users\rjeffrey003\Documents\Work\1 - Weather Data\main.jl:211
in expression starting at c:\Users\rjeffrey003\Documents\Work\1 - Weather Data\main.jl:211

Activity

  1. quinnj-bot commented on Nov 18, 2026

    @quinnj-bot
    Contributor

    This is out of scope for ODBC.load, which appends rows. To insert only new rows, load into a staging table and then run MERGE, or INSERT ... SELECT ... WHERE NOT EXISTS (...), against the target table with DBInterface.execute. Closing as won't-do.

    [comment generated via AI]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions