New capture integration keeps stopping live graph on safari browsers

[waridrox]

What happened just before the problem occurred

The live graph is supposed to work on the new interface like https://publiclab.github.io/spectral-workbench.js/examples/new-capture/ but unfortunately the preview fails on both /v2 and legacy.

The video capture feed is also paused but the waterfall that captures a few seconds of the image works as intended.

Screen.Recording.2021-06-25.at.9.54.02.PM.mov

CC: @jywarren @pydevsg @RuthNjeri

Arose due to #660, later tested at #661

[RuthNjeri]

Referencing this comment as a potential fix #660 (comment)

[jywarren]

I also noticed that the OLD capture graph doesn't work in Safari. I think it may just be an issue with flot? Can we find the line of code where we are refreshing the graph, and see what may be preventing it from firing?

[jywarren]

It could be related to Flot being so old (or a flot safari isssue? https://github.com/flot/flot/search?q=safari&type=issues), and it's possible the fix is really to upgrade to a newer graphing library. But let's not let it block other progress just yet!

[jywarren]

We could try upgrading the flot version to various more recent versions to see if it works.

[waridrox]

Hi @jywarren, so I tried to debug the problem and the issue turned out to be really a weird one 😵

The graph doesn't work on safari but after removing all the stepper styles, the graph seems to be working fine:

working.fine.without.stepper.mp4

The issue though is that after a certain height on the webpage, the live graph stops suddenly and resumes back as it reaches the working height again 🤯:

THE.ISSUE.mp4

Hence I tried to re-implement the stepper functionality but without the stepper (I'm hiding contents with the help of jquery) since the stepper was only hiding the stepwise content in the first place and I have this so far...Though this doesn't look that great due to lack of styles, I wanted to know your thoughts before progressing further:

proposed.fix.mp4

Could we use this as our new capture interface...or should I progress without this and come back later?...thanks!!

[jywarren]

This is especially strange because it is actually working in https://publiclab.github.io/spectral-workbench.js/examples/new-capture/ in safari! What's the difference between that and within the Rails app?

[jywarren]

I think we have to compare this code here:

spectral-workbench/app/views/capture/index2.html.erb

Lines 310 to 360 in 8f89e0f

	var imageData;
	$("#plot-trigger").on('click', function (params) {
	stepper.to(4);
	stepper.to(4); //is this even needed here!!!
	$("#capture-trigger").prop('disabled', true);
	$("#setting-trigger").prop('disabled', true);
	$("#landing-page-trigger").prop('disabled', true);
	$W.saveSpectrum();
	imageData = $W.full_data;
	console.log('what is my image data like', JSON.stringify($W.full_data));
	graph = new SpectralWorkbench.Graph({
	selector: '#graph1' // defaults to "#graph" using jQuery selector syntax
	});
	var imageDataFormatted = [];
	imageData.forEach((item, index) => {
	var formattedItem = [index, item[item.length-1]]
	imageDataFormatted.push(formattedItem)
	})
	var spectrum = new SpectralWorkbench.Spectrum(imageDataFormatted, graph);
	// console.log('spectrum', spectrum.getJSON())
	graph.load(spectrum);
	});
	$("#capture-page-next").on('click', function (params) {
	stepper.to(4);
	$("#capture-trigger").prop('disabled', true);
	$("#setting-trigger").prop('disabled', true);
	$("#landing-page-trigger").prop('disabled', true);
	$W.saveSpectrum();
	imageData = $W.full_data;
	console.log('what is my image data like', JSON.stringify($W.full_data));
	graph = new SpectralWorkbench.Graph({
	selector: '#graph1' // defaults to "#graph" using jQuery selector syntax
	});
	var imageDataFormatted = [];
	imageData.forEach((item, index) => {
	var formattedItem = [index, item[item.length-1]]
	imageDataFormatted.push(formattedItem)
	})
	var spectrum = new SpectralWorkbench.Spectrum(imageDataFormatted, graph);
	// console.log('spectrum', spectrum.getJSON())
	graph.load(spectrum);

https://github.com/publiclab/spectral-workbench.js/blob/4f29a95119af0aef7a4a2e131ec4815df475e1a8/examples/new-capture/index.html#L348-L396

and it's not very tidily written in the first place... but i bet there's some subtle difference!

[jywarren]

Hmm, pretty similar: https://gist.github.com/jywarren/d632d8f5910a598a669902709da8d38c/revisions

On what line is it triggering the graph to display?

[jywarren]

spectral-workbench/app/views/capture/index2.html.erb

Line 269 in 8f89e0f

setInterval($W.getRow,100)

And here is where it's actually generating the graph:

https://github.com/publiclab/spectral-workbench.js/blob/4e3c9f65d51d3dacd4a831ab2aebf39181620db8/examples/capture/getRow.js#L113

[jywarren]

Try inserting a comment and moving it around to see where the breakdown is!

[jywarren]

This is really the key section where it fetches the image data, then creates the data to be graphed:

https://github.com/publiclab/spectral-workbench.js/blob/4e3c9f65d51d3dacd4a831ab2aebf39181620db8/examples/capture/getRow.js#L3-L46

I hope this helps!!

[waridrox]

@jywarren, so I tried to see whether the graph stops or not after moving on the capture section, and the the graph is working as expected:

logs.mov

So this means that this is not a problem arising from the capture code...

[waridrox]

I tried to do play with the height on the working version of sw.js and there also I noticed that the graph was stopping if we tamper with it's height:

not.working.on.sw.js.interface.mov

[waridrox]

One more observation is that the working sw.js version is result of all layers being displayed simultaneously on html page as oppose to different changing layers on /v2 version:

all.layers.on.sw.js.mov

So as soon as the row adjusting layer is gone from the stepper, the graph STOPS:

layer.changing.on.v2.mov

[jywarren]

whoaaaa so all layers being displayed simultaneously -- how is this different between the two? You're not just saying they're CSS display:none but some are actually gone, how?

What accounts for the difference?

Also omg what is this "LAYERS" display - is that a new part of Chrome dev tools??? So cool!

[jywarren]

Excellent use of that tool to narrow in on this, btw. I'm very curious now why exactly the two are different in the first place!

[waridrox]

What accounts for the difference?

Actually that's the intended nature of the bootstrap-stepper...to render the steps like a different page altogether. For example, if someone clicks on step 1 then that defined stepper section should load as a page and be displayed.

But it seems that the sw.js example at http://publiclab.github.io/spectral-workbench.js/examples/new-capture/ isn't following this approach and hence it is able to get away with the live capture actually working 😅.

On the other hand, the desired function of the stepper that works on localhost:3000/capture/v2 takes into effect which removes the rendered view of the camera feed from the layers thereby leading to a failure.

Also omg what is this "LAYERS" display - is that a new part of Chrome dev tools??? So cool!

I'm not sure in which version they introduced it on chrome but chrome does have it under the Layers section though I'm using the web inspector tool on safari.

It is really interesting to see so much going on behind a website 😆-

Screen.Recording.2021-07-16.mov

[waridrox]

So @jywarren, in-short in order to get the interface to work on safari, the graph needs to be in close vicinity of the normal live camera feed which may be either visible or hidden behind the layers...

And here's exactly what I've tried to do...

testing.mov

I'm actually hiding the normal video feed behind the cross section interface in the top left corner of the screen. What are your thoughts on this ?

[jywarren]

But it seems that the sw.js example at http://publiclab.github.io/spectral-workbench.js/examples/new-capture/ isn't following this approach and hence it is able to get away with the live capture actually working 😅.

Hmm, can you point me to the exact lines that are different in your version vs. that one?

I'm actually hiding the normal video feed behind the cross section interface in the top left corner of the screen. What are your thoughts on this ?

here too, can i browse your code solution? Thank you, i think this could work but want to get a close look at the PR, and possibly write a test around it? Thank you!!

[waridrox]

Hi @jywarren, I'm unable to find any dissimilarities between the spectral-workbench's interface code and the new capture interface at /v2, that's because I started with the sw.js interface and began refactoring from there, there is nothing uncommon between the two I'm afraid 🤔...

sw.js interface - https://github.com/publiclab/spectral-workbench.js/blob/main/examples/new-capture/index.html

/v2 interface - https://github.com/waridrox/spectral-workbench/blob/995079cb89ab58e3df0caa68f6f512d86170ba02/app/views/capture/index2.html.erb

---

working.with.stepper.mov

The changes for the proposed version have been pushed to #672 (https://github.com/publiclab/spectral-workbench/blob/31938142cb698c61c490950ee1f6763041f3fa8c/app/views/capture/index2.html.erb), here are the lines through which I am manipulating the graph...

spectral-workbench/app/views/capture/index2.html.erb

Lines 111 to 116 in 3193814

	$("#spectrum-ex").hide();
	$(".capture-settings-hide").hide();
	$("#capture-page-content").width(1).height(1);
	$("#webcam").css('pointer-events', 'none');
	$("#webcam").css('opacity', '0.0');

Can we use this for the time being and proceed forward with the tests or a should I continue on with the old implementation...

[jywarren]

I see your code in #672 and will reply over there!