Stipple ZMQ and realtime update of dataframes

[ghost]

Hi all
I just found out about Stipple and considering changing from looking into Dash for my usecase. So far I am VERY impressed with Stipple as it just makes sense the way it is laid out and it's plan for the future.

I have a concern about Stipple.jl current ability to handle realtime update of a dataframe.

Here is my usecase.

I open a ZMQ pipe and receive a STREAM ( 1 - 10 every second) of messages
the dataframe will have 100 rows and 20 columns. The row and column are in the message together with the VALUE to be used to update.

each message contains information that will be used to UPDATE a value in a dataframe by locating the row and value to be updated.
so the dataframe is a realtime dashboard.

In dash this is probably handled by callbacks but haven't tried it out just yet as I am looking into Stipple.jl.

I just want to know if this is something that I can do NOW or will it be something in the future.

[essenciary]

Yes, you can do it now with DataFrames and StippleUI.Tables.
It should also be possible to update individual cells instead of sending over the whole data table. @hhaensel has built this layer very recently - Helmut, can you provide more insights re atomic updates?

[ghost]

excellent news!!! well done @hhaensel here's a clarification of my situation, thanks to your EXCELLENT approach I might just drop the column creation approach as I created the dataframe web page in MINUTES!!! OUTSTANDING

the message comes in

"top~price~12.23"  

is split up and used as update coords. in this case if the row or the column doesn't exist then it's added.

global source,sym_in,field_in ,value_in  = split( message , "~")

           global value_fl = parse(Float64, value_in) # convert to Float64

           try
              global field_out = zmq_dash[ field_in]                # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"

              println("message to add : ", source," ",sym_in," ",field_out," " ,value_in)

           catch e

              println("field_in : ", field_in, " not in cols" )

           end

           if sym_in in df_dash_table[!,:sym]

              df_dash_table[findfirst(==(sym_in),df_dash_table.sym),findfirst(==(field_out),names(df_dash_table))] = value_fl
         
           else

               push!(df_dash_table, (sym_in, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0 ) )
               df_dash_table[findfirst(==(sym_in),df_dash_table.sym),findfirst(==(field_out),names(df_dash_table))] = value_fl

           end 

[hhaensel]

Well, if you change your dataframe not only by appending but by inserting, atomic update won't help. Moreover, the data rendering of tables is a bit more complex than updating dicts or arrays.
So no good clue for atomic updates.
What you could do, however, is to update your dataframe silently and Send a notify every second, or whatever a reasonable updaterate is.

If your df_dash_table is directly the dataframe in your datatzble field. The table will anyhow be up to date and you only need to do

notify(model.datatable)

Does that make sense to you?

[ghost]

Hi there @hhaensel and @essenciary
Sorry but I can't get what you suggest working. Thanks for the help but I'm probably going back to dash. Shame but necessary now. I tried to get more guidance ( see below) and tried on julia discourse. I just ran out of time. Wish you well on this project it will be successful.
theakson

[ghost]

hi there
not sure what I am doing wrong. I took your advice and added

notify(TontineModel.tontine_data )

to update my dataframe in the UI model. I get the error

Received request: STK~WDAY~IV~900.0
message to add : STK WDAY iv 900.0
ERROR: LoadError: type DataType has no field tontine_data
Stacktrace:
 [1] getproperty
   @ ./Base.jl:37 [inlined]
 [2] top-level scope
   @ ~/tontine_2022/just_messing/7_18_22_zmq_stpp_df_working.jl:115
in expression starting at /home/dave/tontine_2022/just_messing/7_18_22_zmq_stpp_df_working.jl:78
dave@deepthought:~/tontine_2022/just_messing$ 

in the code below. What should I be using please?

using Stipple
using StippleUI
using StipplePlotly

using CSV, DataFrames, Dates
using ZMQ

#=
find all LISTEN ports
sudo lsof -i -P -n | grep LISTEN

julia      336369            dave   30u  IPv4 87668874      0t0  TCP *:5555 (LISTEN)
julia      336369            dave   31u  IPv4 87668875      0t0  TCP *:5556 (LISTEN)

kill a pid

sudo kill -9 336369

=#

# df_data = CSV.File( "/home/dave/tontine_2022/csv/tont_test_data.csv") |> DataFrame


dash_columns = ["sym","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","volume"]
 
df_table = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns ])


zmq_dash = Dict("LAST" => "price","CLOSE" => "price","OPTION_IMPLIED_VOL" => "iv",
                         "VOLUME"  => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
                         "HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile")





@reactive mutable struct TontineModel <: ReactiveModel
  tontine_data::R{DataTable} = DataTable(df_table)   
end


function ui(model::TontineModel)
  page(
    model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
    prepend = style(
    ),

    [
      heading("heading Tontine2 7/18/22")

      row([
        cell(class="st-module", [
          h5("h5 tontine data")
          table(:tontine_data)
        ])
      ])
    ]
  )
end

route("/") do
  TontineModel |> init |> ui |> html
end

up(9000; async = true, server = Stipple.bootstrap())


context = Context()

socket = Socket(context, PULL)

ZMQ.bind(socket, "tcp://*:5555")

println("starting IN Socket 5555")

println("entering main, waiting for message")

while true
             
   message = String(ZMQ.recv(socket))
   println("Received request: $message")
   if message == "END"
      println("dying")
      ZMQ.close(socket)
      ZMQ.close(context)
      break
   end

   global in_source,sym_in,field_in ,value_in  = split( message , "~")

   global value_fl = parse(Float64, value_in) # convert to Float64

   try
     global field_out = zmq_dash[ field_in]                # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"

     println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)

   catch e

     println("field_in : ", field_in, " not in cols" )

   end

   if sym_in in df_table[!,:sym]

     df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   else

      push!(df_table, (sym_in, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0 ) )
      df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   end 

   notify(TontineModel.tontine_data )
  # println(df_table)
end





#=  ZMQ SERVER ( RECV MSG = REP  CLIENT (SEND MSG)  = REQ 

https://discourse.julialang.org/t/maintain-connection-in-zmq/34243

using ZMQ

function server_listen(server)
   #setproperty!(s1, :set_tcp_keepalive, 1)
    while true
       msg = recv(server, String)
       println("Server received: ", msg)
       send(server, "Client request processed.")
   end
end

function client_send(client)
   for i in 1:1000
      send(client, string("Client request # ", i))
      msgin = recv(client, String)
      println("Request status: ", msgin)
   end
end

# initialize connections
ctx = Context()
server = Socket(ctx, REP)
bind(server, "tcp://127.0.0.1:5555")
client = Socket(ctx, REQ)
connect(client, "tcp://127.0.0.1:5555")

# start server
@async server_listen(server)

# stress test
[client_send(client) for _ in 1:5]
close(server)
close(client)

=#

[hhaensel]

You have no handler installed and no global model defined, so your code will not work.
You could do

route("/") do
    global model
    model = TontineModel |> init |> ui |> html
end

[...]

notify(model.tontine_data)

I'm not sure how fast your ZMQ update is, maybe you have to supply an interval...

[ghost]

Hi there @hhaensel
thank you for taking the time to help a noob. I posted a cutback version of the code to highlight the problem and that was a mistake, sorry for the confusion.
If you look and my reply to your original solution #126 (comment) you'll see I tried what you suggest and got an error. see the commented out code.

That's why I'm stuck. If you run the stipple code then in another terminal run the test message code you'll see what I mean. The refresh interval will be 1 - 2 seconds for the message queue. Any help would be appreciated.

here's the code to send some messages to the stipple code

using ZMQ
using CSV
using DataFrames



#=
find all LISTEN ports
sudo lsof -i -P -n | grep LISTEN

julia      336369            dave   30u  IPv4 87668874      0t0  TCP *:5555 (LISTEN)
julia      336369            dave   31u  IPv4 87668875      0t0  TCP *:5556 (LISTEN)

kill a pid

sudo kill -9 336369

=#




context = Context()

# Socket to talk to server
println(" opening 5555 ")
socket = Socket(context, PUSH)
ZMQ.connect(socket, "tcp://localhost:5555")




#=
https://discourse.julialang.org/t/how-to-select-rows-from-a-dataframe-and-then-create-a-dataframe-with-multiple-columns-from-the-selection/77652

In general data frame indexing has the following form:

source_data_frame[row_selector, column_selector]
=#

# symbols = ["AAPL","QCOM","JPM","GOOG"]







fields  = ["LAST","OPTION_IMPLIED_VOL","VOLUME","IV"]


function get_expert_symbols(csv_file)

    df_symbol_to_process_list = CSV.read( csv_file , DataFrame; delim=",", header=true) #, skipto=2, header=true)
    
    replace!(df_symbol_to_process_list.ib_symbol, " " => missing)

    df_cleaned = filter(:ib_symbol => x -> !(ismissing(x) || isnothing(x)) , df_symbol_to_process_list)
   
    return df_cleaned
 
end




function generate_TWS_messages()
 

        for test in 1:1000

            source = "STK"

            value = round((rand( (2, 9))* 100);digits = 2)

            symbol = rand(symbols.Sym)

            field = rand(fields)

            send_message = (source * "~" * symbol * "~" * field * "~"* string(value) )
                    
            # send_message = test_new_message(symbols)

            println(send_message)

            ZMQ.send(socket, send_message)             #"Hello" )
        end
end

symbol_data = get_expert_symbols("/home/dave/tontine_2022/csv/tontine_symbols_jl.csv" )

symbols  = symbol_data[symbol_data.Type .== "STK", ["Sym", "exchange", "Country"]]

# println(symbols)

generate_TWS_messages()


ZMQ.send(socket,"END")

# Making a clean exit.
ZMQ.close(socket)
ZMQ.close(context)

and here's my stipple code

using Stipple
using StippleUI
using StipplePlotly

using CSV, DataFrames, Dates
using ZMQ

#=
find all LISTEN ports
sudo lsof -i -P -n | grep LISTEN

julia      336369            dave   30u  IPv4 87668874      0t0  TCP *:5555 (LISTEN)
julia      336369            dave   31u  IPv4 87668875      0t0  TCP *:5556 (LISTEN)

kill a pid

sudo kill -9 336369

=#

# df_data = CSV.File( "/home/dave/tontine_2022/csv/tont_test_data.csv") |> DataFrame


dash_columns = ["sym","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","volume"]
 
df_table = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns ])


zmq_dash = Dict("LAST" => "price","CLOSE" => "price","OPTION_IMPLIED_VOL" => "iv",
                         "VOLUME"  => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
                         "HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile")





@reactive mutable struct TontineModel <: ReactiveModel
  tontine_data::R{DataTable} = DataTable(df_table)   
end


function ui(model::TontineModel)
  page(
    model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
    prepend = style(
    ),

    [
      heading("heading Tontine2 7/18/22")

      row([
        cell(class="st-module", [
          h5("h5 tontine data")
          table(:tontine_data)
        ])
      ])
    ]
  )
end

route("/") do
  TontineModel |> init |> ui |> html
end

up(9000; async = true, server = Stipple.bootstrap())


context = Context()

socket = Socket(context, PULL)

ZMQ.bind(socket, "tcp://*:5555")

println("starting IN Socket 5555")

println("entering main, waiting for message")

while true
             
   message = String(ZMQ.recv(socket))
   println("Received request: $message")
   if message == "END"
      println("dying")
      ZMQ.close(socket)
      ZMQ.close(context)
      break
   end

   global in_source,sym_in,field_in ,value_in  = split( message , "~")

   global value_fl = parse(Float64, value_in) # convert to Float64

   try
     global field_out = zmq_dash[ field_in]                # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"

     println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)

   catch e

     println("field_in : ", field_in, " not in cols" )

   end

   if sym_in in df_table[!,:sym]

     df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   else

      push!(df_table, (sym_in, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0 ) )
      df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   end 

#from https://youtu.be/xlieoONTLUs?t=2301

  #notify(TontineModel.tontine_data ) 

   Observables.notify!(TontineModel.tontine_data)





end





#=  ZMQ SERVER ( RECV MSG = REP  CLIENT (SEND MSG)  = REQ 

https://discourse.julialang.org/t/maintain-connection-in-zmq/34243

using ZMQ

function server_listen(server)
   #setproperty!(s1, :set_tcp_keepalive, 1)
    while true
       msg = recv(server, String)
       println("Server received: ", msg)
       send(server, "Client request processed.")
   end
end

function client_send(client)
   for i in 1:1000
      send(client, string("Client request # ", i))
      msgin = recv(client, String)
      println("Request status: ", msgin)
   end
end

[hhaensel]

I am afraid you didn't understand my answer.
Your route definition should be

route("/") do
  global model
  model = TontineModel |> init |> ui |> html
end

and your main code block should contain

global model
model.isready[] && notify(model.tontine_data)

The model.isready[] && prevents that the model is updated without a connected client.

That's not the setup I would chose, because with this setup only one client can connect and will be served the updates.
But to check, whether you are ok with the view and the appearance of a Julia app, that's what I would test first.

[comicpilsen]

hi @hhaensel
I'm not at the machine right now BUT if this works I apologize. It's not that I didn't understand your answer I just didn't read it properly. I completely missed the global. Coming from Python this global approach keeps tripping me up but I'll get used to it in the end. Thank you for sticking with a noob who CLEARLY has lost the ability to read AND process simple concepts like scope. Have an excellent weekend.
theakson
ps sorry again..

[hhaensel]

No problem. Looking forward to your testing results 😃

[ghost]

@hhaensel
Sorry. no luck. I'm just not getting the global definition. Am I putting the code you suggested in the right place ? I keep getting

┌ Info: 
└ Web Server starting at http://127.0.0.1:9000 
starting IN Socket 5555
entering main, waiting for message
ERROR: LoadError: UndefVarError: model not defined
Stacktrace:
 [1] top-level scope
   @ ~/tontine_2022/just_messing/7_29_22_zmq_stpp_df_working.jl:70
in expression starting at /home/dave/tontine_2022/just_messing/7_29_22_zmq_stpp_df_working.jl:66
dave@deepthought:~/tontine_2022/just_messing$ 

from this code

using Stipple
using StippleUI
using StipplePlotly

using CSV, DataFrames, Dates
using ZMQ

dash_columns = ["sym","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","volume"]
 
df_table = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns ])


zmq_dash = Dict("LAST" => "price","CLOSE" => "price","OPTION_IMPLIED_VOL" => "iv",
                         "VOLUME"  => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
                         "HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile")





@reactive mutable struct TontineModel <: ReactiveModel
  tontine_data::R{DataTable} = DataTable(df_table)   
end


function ui(model::TontineModel)
  page(
    model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
    prepend = style(
    ),

    [
      heading("heading Tontine2 7/18/22")

      row([
        cell(class="st-module", [
          h5("h5 tontine data")
          table(:tontine_data)
        ])
      ])
    ]
  )
end

route("/") do
  global model
  model = TontineModel |> init |> ui |> html
end

up(9000; async = true, server = Stipple.bootstrap())


context = Context()

socket = Socket(context, PULL)

ZMQ.bind(socket, "tcp://*:5555")

println("starting IN Socket 5555")

println("entering main, waiting for message")




while true

   global model

   model.isready[] && notify(model.tontine_data) # prevents that the model is updated without a connected client.

   notify(model.tontine_data)
 
                
   message = String(ZMQ.recv(socket))

   println("Received request: $message")
   if message == "END"
      println("dying")
      ZMQ.close(socket)
      ZMQ.close(context)
      break
   end

   global in_source,sym_in,field_in ,value_in  = split( message , "~")

   global value_fl = parse(Float64, value_in) # convert to Float64

   try
     global field_out = zmq_dash[ field_in]                # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"

     println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)

   catch e

     println("field_in : ", field_in, " not in cols" )

   end

   if sym_in in df_table[!,:sym]

     df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   else

      push!(df_table, (sym_in, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0 ) )
      df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   end 

  
  
  # println(df_table)

end

[hhaensel]

Sorry, I was too quick on my mobile.
Try placing a

model = TontineModel |> init

after the global declaration and replace the route by

  model |> ui |> html

[hhaensel]

In that version all clients connect to the same model.

[ghost]

@hhaensel
first of all I feel REALLY bad that you are doing all of this on a mobile, that CANNOT be fun and I do appreciate it. BUT I'm a little confused now. There are two global declarations for model in the code and I suspect that's because I thought I had an idea about how the scope was working and the manner in which the model was initialized. I'm headed down a rabbit hole here.

I THINK you are saying the code should look like this now and I am a little baffled as to what that means.

using Stipple
using StippleUI
using StipplePlotly

using CSV, DataFrames, Dates
using ZMQ

dash_columns = ["sym","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","volume"]
 
df_table = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns ])


zmq_dash = Dict("LAST" => "price","CLOSE" => "price","OPTION_IMPLIED_VOL" => "iv",
                         "VOLUME"  => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
                         "HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile")





@reactive mutable struct TontineModel <: ReactiveModel
  tontine_data::R{DataTable} = DataTable(df_table)   
end


function ui(model::TontineModel)
  page(
    model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
    prepend = style(
    ),

    [
      heading("heading Tontine2 7/18/22")

      row([
        cell(class="st-module", [
          h5("h5 tontine data")
          table(:tontine_data)
        ])
      ])
    ]
  )
end

route("/") do
  global model

  model |> ui |> html

 
  # old model = TontineModel |> init |> ui |> html
end

up(9000; async = true, server = Stipple.bootstrap())


context = Context()

socket = Socket(context, PULL)

ZMQ.bind(socket, "tcp://*:5555")

println("starting IN Socket 5555")

println("entering main, waiting for message")

model = TontineModel |> init


while true

   global model

   model.isready[] && notify(model.tontine_data) # prevents that the model is updated without a connected client.

   notify(model.tontine_data)
 
                
   message = String(ZMQ.recv(socket))

   println("Received request: $message")
   if message == "END"
      println("dying")
      ZMQ.close(socket)
      ZMQ.close(context)
      break
   end

   global in_source,sym_in,field_in ,value_in  = split( message , "~")

   global value_fl = parse(Float64, value_in) # convert to Float64

   try
     global field_out = zmq_dash[ field_in]                # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"

     println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)

   catch e

     println("field_in : ", field_in, " not in cols" )

   end

   if sym_in in df_table[!,:sym]

     df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   else

      push!(df_table, (sym_in, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0 ) )
      df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   end 

  
  
  # println(df_table)

end

[hhaensel]

Well let me try, I finally got some time to boot up my computer ...

using Stipple
using StippleUI
using StipplePlotly

using CSV, DataFrames, Dates
using ZMQ

dash_columns = ["sym","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","volume"]
 
df_table = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns ])

zmq_dash = Dict("LAST" => "price","CLOSE" => "price","OPTION_IMPLIED_VOL" => "iv",
                         "VOLUME"  => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
                         "HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile")


@reactive mutable struct TontineModel <: ReactiveModel
  tontine_data::R{DataTable} = DataTable(df_table)   
end


function ui(model::TontineModel)
  page(
    model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
    prepend = style(
    ),

    [
      heading("heading Tontine2 7/18/22")

      row([
        cell(class="st-module", [
          h5("h5 tontine data")
          table(:tontine_data)
        ])
      ])
    ]
  )
end

model = TontineModel |> init

route("/") do
  global model
  model  |> ui |> html
end

up(9000; async = true, server = Stipple.bootstrap())


context = Context()

socket = Socket(context, PULL)

ZMQ.bind(socket, "tcp://*:5555")

println("starting IN Socket 5555")

println("entering main, waiting for message")


while true

   message = String(ZMQ.recv(socket))

   println("Received request: $message")
   if message == "END"
      println("dying")
      ZMQ.close(socket)
      ZMQ.close(context)
      break
   end

   global in_source, sym_in, field_in , value_in = split( message , "~")

   global value_fl = parse(Float64, value_in) # convert to Float64

   try
     global field_out = zmq_dash[ field_in]                # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"

     println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)

   catch e

     println("field_in : ", field_in, " not in cols" )

   end

   if sym_in in df_table[!,:sym]

     df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   else

      push!(df_table, (sym_in, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0 ) )
      df_table[findfirst(==(sym_in),df_table.sym),findfirst(==(field_out),names(df_table))] = value_fl

   end 

   model.isready[] && notify(model.tontine_data) # prevents that the model is updated without a connected client.
  
  # println(df_table)

end

should most probably work

[ghost]

perfect! thank you so much for all your VERY kind help. I now have a model to play with and work through the logic. This is REALLY useful to a noob like me who lost the ability to read. My "mates" are making me wear a straw boater for the whole day monday because of this. Thanks again for such a wonderful package and helping me out.
theakson

[hhaensel]

Happy that this worked :-)

@essenciary, @AbhimanyuAryan
I think this is a typical use case that we have not yet covered in our StippleDemos.

What would you think about the following approach?

using Stipple
using StippleUI
using StipplePlotly

using CSV, DataFrames, Dates

dash_columns = ["sym","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","volume"]

df = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns])
df_table = Observable(df)

zmq_dash = Dict("LAST" => "price","CLOSE" => "price","OPTION_IMPLIED_VOL" => "iv",
                         "VOLUME"  => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
                         "HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile")

@reactive mutable struct TontineModel <: ReactiveModel
    tontine_data::R{DataTable} = DataTable(df_table[])   
end

function ui(model::TontineModel)
    page(
        model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),

        [
        heading("heading Tontine2 7/18/22")

        row([
            cell(class="st-module", [
            h5("h5 tontine data")
            table(:tontine_data)
            ])
        ])
        ]
    )
end

function handlers(model)
    on(df_table) do _
        notify(model.tontine_data)
    end

    model
end

route("/") do
    TontineModel |> init |> handlers |> ui |> html
end

# up(9000; async = true, server = Stipple.bootstrap())
up()


for _ in 1:10
    # generate a random entry
    message = join([
        ["A", "B", "C", "D", "E", "F", "G"][rand(1:7)],
        ["a", "b", "c", "d", "e", "f", "g"][rand(1:7)],
        collect(keys(zmq_dash))[rand(1:length(zmq_dash))],
        string(rand())
    ], "~")
    sleep(0.5)

    global in_source, sym_in, field_in, value_in = split(message , "~")
    global value_fl = parse(Float64, value_in) # convert to Float64

    try
        global field_out = zmq_dash[field_in]                # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"
        println("message to add : $in_source, $sym_in, $field_out, $value_in")
    catch e
        println("field_in : ", field_in, " not in cols" )
    end

    sym_in in df_table[][!,:sym] || push!(df_table[], (sym_in, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0))
    df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==(field_out), names(df_table[]))] = value_fl
    notify(df_table)
end

I replaced the ZMQ by a random generator ...

[hhaensel]

It makes df_table an Observable and attaches a listener on each new model. That way all models would receive the same updates, but all other stuff would be client dependent.
EDIT: And they would all share the same variable and not waste any memory ...

[ghost]

@hhaensel @essenciary
I LOVE it !!! if I'd had this 12 days ago I wouldn't be walking around on Monday wearing the straw boater. I like handlers functionality. I think that it would be an excellent addition to the stipple usecase library. Seeing the data populate on screen is very satisfying and leads into all sorts of other scenarios. Excellent idea.