Architect Karl Kaellsceny designed the palace in early historicism style for the Jirkovsky family in 1874. The building originally housed the inn K crnom konju (Black Horse Inn) and was occasionally used as a theatre venue. Count Eltz bought the palace around 1890 and added a smaller two-storey building featuring identically shaped facades to its right side in the late 1800s. This building was later adapted to serve as the seat of the Financial Directorate, and more recently of the Secondary School of Economics. The Lavoslav Ružička Polytechnic was located there from 2005 to 2023. The building was heavily damaged during the Croatian Homeland War and its reconstruction was financed with a donation from the Dubrovnik-Neretva County.