Hi everyone, Can I get your input on the following questions : - What are your best practices and recommendations for developing and testing concurrent software? - How to discover need for synchronization/critical sections/ when doing TDD? - How to write code to avoid dead-locks? Thanks for your help, Noury
This was useful in doing our single threaded JavaScript to Multi-threaded Swift IOS platform code [http://reactivex.io](http://reactivex.io/) .... John M. McIntosh. Corporate Smalltalk Consulting Ltd https://www.linkedin.com/in/smalltalk Sent from ProtonMail Mobile On Wed, Sep 4, 2019 at 15:32, N. Bouraqadi <bouraqadi@gmail.com> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
Not to sound flippant - but do everything you can to limit where you use concurrency (and if possible, not at all). It adds lots of complication that often isn’t necessary. Where you do need it, queue results or commands and then have workers to process those - workers, queues and commands put you back into easily testable territory. Then you are left with the concurrent bit - hopefully now quite tiny and specific, which you you can normally blast and test if you have concurrency hole - but in the realms of can it queue things up in an orderly fashion and dequeue them as well. Deep code inspection on this is also helpful (albeit manual). Tim Sent from my iPhone
On 4 Sep 2019, at 14:55, John M McIntosh <johnmci@smalltalkconsulting.com> wrote:
This was useful in doing our single threaded JavaScript to Multi-threaded Swift IOS platform code http://reactivex.io
.... John M. McIntosh. Corporate Smalltalk Consulting Ltd https://www.linkedin.com/in/smalltalk
Sent from ProtonMail Mobile
On Wed, Sep 4, 2019 at 15:32, N. Bouraqadi <bouraqadi@gmail.com> wrote: Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
Precisely my thinking! try to avoid concurrency as much as possible and if you _really_ have to use it, try a queue approach. Georg Heeg eK Wallstraße 22 06366 Köthen Tel.: 03496/214328 FAX: 03496/214712 Amtsgericht Dortmund HRA 12812 Am 5. September 2019 um 09:34:25, Tim Mackinnon (tim@testit.works<mailto:tim@testit.works>) schrieb: Not to sound flippant - but do everything you can to limit where you use concurrency (and if possible, not at all). It adds lots of complication that often isn’t necessary. Where you do need it, queue results or commands and then have workers to process those - workers, queues and commands put you back into easily testable territory. Then you are left with the concurrent bit - hopefully now quite tiny and specific, which you you can normally blast and test if you have concurrency hole - but in the realms of can it queue things up in an orderly fashion and dequeue them as well. Deep code inspection on this is also helpful (albeit manual). Tim Sent from my iPhone On 4 Sep 2019, at 14:55, John M McIntosh <johnmci@smalltalkconsulting.com<mailto:johnmci@smalltalkconsulting.com>> wrote: This was useful in doing our single threaded JavaScript to Multi-threaded Swift IOS platform code http://reactivex.io<http://reactivex.io/> .... John M. McIntosh. Corporate Smalltalk Consulting Ltd https://www.linkedin.com/in/smalltalk Sent from ProtonMail Mobile On Wed, Sep 4, 2019 at 15:32, N. Bouraqadi <bouraqadi@gmail.com<mailto:bouraqadi@gmail.com>> wrote: Hi everyone, Can I get your input on the following questions : - What are your best practices and recommendations for developing and testing concurrent software? - How to discover need for synchronization/critical sections/ when doing TDD? - How to write code to avoid dead-locks? Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org<mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org<mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
Hello, Supposing you are working on a desktop app and you need to keep your main image responsive and want to offload things to a worker image on a separate processor you can hook it to the keyboard processor once finished. If you are basically making a clone of your image it is easy to make a switch (dev/runtime) to do all the development and debugging in the same image. This is at least my scenario. The main application works on a read-write sqlite file. The worker makes a read only copy , slowly synchronises everything with Postgres on compose.io <http://compose.io/>, when finished puts itself on the keyboard queue and when it his turn writes the differences to the local sqlite work file and signals an UI update. The keyboard hook makes it possible for the worker to come back on the main thread as a normal user interaction. Hope this helps, Maarten,
Le 5 sept. 2019 à 10:27, Karsten Kusche <karsten@heeg.de> a écrit :
Precisely my thinking! try to avoid concurrency as much as possible and if you _really_ have to use it, try a queue approach.
Georg Heeg eK Wallstraße 22 06366 Köthen
Tel.: 03496/214328 FAX: 03496/214712 Amtsgericht Dortmund HRA 12812
Am 5. September 2019 um 09:34:25, Tim Mackinnon (tim@testit.works <mailto:tim@testit.works>) schrieb:
Not to sound flippant - but do everything you can to limit where you use concurrency (and if possible, not at all). It adds lots of complication that often isn’t necessary.
Where you do need it, queue results or commands and then have workers to process those - workers, queues and commands put you back into easily testable territory.
Then you are left with the concurrent bit - hopefully now quite tiny and specific, which you you can normally blast and test if you have concurrency hole - but in the realms of can it queue things up in an orderly fashion and dequeue them as well. Deep code inspection on this is also helpful (albeit manual).
Tim
Sent from my iPhone
On 4 Sep 2019, at 14:55, John M McIntosh <johnmci@smalltalkconsulting.com <mailto:johnmci@smalltalkconsulting.com>> wrote:
This was useful in doing our single threaded JavaScript to Multi-threaded Swift IOS platform code http://reactivex.io <http://reactivex.io/>
.... John M. McIntosh. Corporate Smalltalk Consulting Ltd https://www.linkedin.com/in/smalltalk <https://www.linkedin.com/in/smalltalk>
Sent from ProtonMail Mobile
On Wed, Sep 4, 2019 at 15:32, N. Bouraqadi <bouraqadi@gmail.com <mailto:bouraqadi@gmail.com>> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org>
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org>
Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org>
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org>
Adding to my previous speakers … At a (non-Smalltalk) conference I once watched a very comprehensive and good presentation by a Professor who also worked for companies. So he had both the theoretical and the practical background and he covered a few different uses cases. The closing summary of his presentation was: - Concurrency is a must (in the light of the end of Moore’s Law) - Avoid concurrency as long as you can - If you really need concurrency then there is no one-size-fits-all solution: each performance/scaling problem is different and needs a different approach I am sorry I don’t remember the neither the name of the Professor nor the name of the conference ☹ Yet if I do and find the presentation on the web I will distribute the link here. Cheers Helge Helge Nowak Cincom Smalltalk Technical Account Manager [http://internal.cincom.com/signature/logo.png]<http://www.cincomsmalltalk.com/> Cincom Systems GmbH & Co. oHG Humboldtstraße 3 60318 Frankfurt am Main GERMANY office mobile website email +49 89 89 66 44 94 +49 172 74 00 402 http://www.cincomsmalltalk.com hnowak@cincom.com A standpoint is an intellectual horizon of radius zero. -- Albert Einstein Geschäftsführer/Managing Directors: Thomas M. Nies, Donald E. Vick oHG mit Sitz/based in Frankfurt am Main (Amtsgericht Frankfurt am Main HRA 50297) Pers. haftender Gesellschafter/Partner liable to unlimited extent: Cincom Systems Verwaltungsgesellschaft mbH (Amtsgericht Königstein/Ts. HRB 5069) --- CONFIDENTIALITY STATEMENT --- This e-mail transmission contains information that is intended to be privileged and confidential. It is intended only for the addressee named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited, please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From: Esug-list <esug-list-bounces@lists.esug.org> On Behalf Of Maarten Mostert Sent: Thursday, 5 September 2019 10:57 To: esug-list@lists.esug.org Subject: Re: [Esug-list] Concurrency Best Practices + Tests Hello, Supposing you are working on a desktop app and you need to keep your main image responsive and want to offload things to a worker image on a separate processor you can hook it to the keyboard processor once finished. If you are basically making a clone of your image it is easy to make a switch (dev/runtime) to do all the development and debugging in the same image. This is at least my scenario. The main application works on a read-write sqlite file. The worker makes a read only copy , slowly synchronises everything with Postgres on compose.io<http://compose.io>, when finished puts itself on the keyboard queue and when it his turn writes the differences to the local sqlite work file and signals an UI update. The keyboard hook makes it possible for the worker to come back on the main thread as a normal user interaction. Hope this helps, Maarten, Le 5 sept. 2019 à 10:27, Karsten Kusche <karsten@heeg.de<mailto:karsten@heeg.de>> a écrit : Precisely my thinking! try to avoid concurrency as much as possible and if you _really_ have to use it, try a queue approach. Georg Heeg eK Wallstraße 22 06366 Köthen Tel.: 03496/214328 FAX: 03496/214712 Amtsgericht Dortmund HRA 12812 Am 5. September 2019 um 09:34:25, Tim Mackinnon (tim@testit.works<mailto:tim@testit.works>) schrieb: Not to sound flippant - but do everything you can to limit where you use concurrency (and if possible, not at all). It adds lots of complication that often isn’t necessary. Where you do need it, queue results or commands and then have workers to process those - workers, queues and commands put you back into easily testable territory. Then you are left with the concurrent bit - hopefully now quite tiny and specific, which you you can normally blast and test if you have concurrency hole - but in the realms of can it queue things up in an orderly fashion and dequeue them as well. Deep code inspection on this is also helpful (albeit manual). Tim Sent from my iPhone On 4 Sep 2019, at 14:55, John M McIntosh <johnmci@smalltalkconsulting.com<mailto:johnmci@smalltalkconsulting.com>> wrote: This was useful in doing our single threaded JavaScript to Multi-threaded Swift IOS platform code http://reactivex.io<http://reactivex.io/> .... John M. McIntosh. Corporate Smalltalk Consulting Ltd https://www.linkedin.com/in/smalltalk Sent from ProtonMail Mobile On Wed, Sep 4, 2019 at 15:32, N. Bouraqadi <bouraqadi@gmail.com<mailto:bouraqadi@gmail.com>> wrote: Hi everyone, Can I get your input on the following questions : - What are your best practices and recommendations for developing and testing concurrent software? - How to discover need for synchronization/critical sections/ when doing TDD? - How to write code to avoid dead-locks? Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org<mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org<mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org<mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org<mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
See Stefan Marr's talks at Smalltalks 2012. On 9/5/19 8:14 , Nowak, Helge wrote:
Adding to my previous speakers …
At a (non-Smalltalk) conference I once watched a very comprehensive and good presentation by a Professor who also worked for companies. So he had both the theoretical and the practical background and he covered a few different uses cases. The closing summary of his presentation was:
- Concurrency is a must (in the light of the end of Moore’s Law)
- Avoid concurrency as long as you can
- If you really need concurrency then there is no one-size-fits-all solution: each performance/scaling problem is different and needs a different approach
I am sorry I don’t remember the neither the name of the Professor nor the name of the conference LYet if I do and find the presentation on the web I will distribute the link here.
Cheers
Helge
*Helge Nowak *Cincom Smalltalk Technical Account Manager**
http://internal.cincom.com/signature/logo.png <http://www.cincomsmalltalk.com/> Cincom Systems GmbH & Co. oHG Humboldtstraße 3 60318 Frankfurt am Main GERMANY
office mobile
website email
+49 89 89 66 44 94 +49 172 74 00 402
http://www.cincomsmalltalk.com hnowak@cincom.com
/A standpoint is an intellectual horizon of radius zero. -- Albert Einstein
/
Geschäftsführer/Managing Directors: Thomas M. Nies, Donald E. Vick oHG mit Sitz/based in Frankfurt am Main (Amtsgericht Frankfurt am Main HRA 50297) Pers. haftender Gesellschafter/Partner liable to unlimited extent: Cincom Systems Verwaltungsgesellschaft mbH (Amtsgericht Königstein/Ts. HRB 5069)
--- CONFIDENTIALITY STATEMENT ---
This e-mail transmission contains information that is intended to be privileged and confidential. It is intended only for the addressee named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited, please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.
*From:*Esug-list <esug-list-bounces@lists.esug.org> *On Behalf Of *Maarten Mostert *Sent:* Thursday, 5 September 2019 10:57 *To:* esug-list@lists.esug.org *Subject:* Re: [Esug-list] Concurrency Best Practices + Tests
Hello,
Supposing you are working on a desktop app and you need to keep your main image responsive and want to offload things to a worker image on a separate processor you can hook it to the keyboard processor once finished.
If you are basically making a clone of your image it is easy to make a switch (dev/runtime) to do all the development and debugging in the same image.
This is at least my scenario. The main application works on a read-write sqlite file. The worker makes a read only copy , slowly synchronises everything with Postgres on compose.io <http://compose.io>, when finished puts itself on the keyboard queue and when it his turn writes the differences to the local sqlite work file and signals an UI update.
The keyboard hook makes it possible for the worker to come back on the main thread as a normal user interaction.
Hope this helps,
Maarten,
Le 5 sept. 2019 à 10:27, Karsten Kusche <karsten@heeg.de <mailto:karsten@heeg.de>> a écrit :
Precisely my thinking! try to avoid concurrency as much as possible and if you _really_ have to use it, try a queue approach.
Georg Heeg eK
Wallstraße 22
06366 Köthen
Tel.: 03496/214328
FAX: 03496/214712
Amtsgericht Dortmund HRA 12812
Am 5. September 2019 um 09:34:25, Tim Mackinnon (tim@testit.works <mailto:tim@testit.works>) schrieb:
Not to sound flippant - but do everything you can to limit where you use concurrency (and if possible, not at all). It adds lots of complication that often isn’t necessary.
Where you do need it, queue results or commands and then have workers to process those - workers, queues and commands put you back into easily testable territory.
Then you are left with the concurrent bit - hopefully now quite tiny and specific, which you you can normally blast and test if you have concurrency hole - but in the realms of can it queue things up in an orderly fashion and dequeue them as well. Deep code inspection on this is also helpful (albeit manual).
Tim
Sent from my iPhone
On 4 Sep 2019, at 14:55, John M McIntosh <johnmci@smalltalkconsulting.com <mailto:johnmci@smalltalkconsulting.com>> wrote:
This was useful in doing our single threaded JavaScript to Multi-threaded Swift IOS platform code http://reactivex.io <http://reactivex.io/>
....
John M. McIntosh. Corporate Smalltalk Consulting Ltd https://www.linkedin.com/in/smalltalk
Sent from ProtonMail Mobile
On Wed, Sep 4, 2019 at 15:32, N. Bouraqadi <bouraqadi@gmail.com <mailto:bouraqadi@gmail.com>> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
TL; DR: generally, developing a good sense of what to do ***before writing a single line of code*** works really well. On 9/4/19 6:32 , N. Bouraqadi wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org .
Hi guys I imagine that noury knows most of the remarks you sent it. He is far from dull. Now it would be nice to read his questions and reply to them and not just tell him “oh do not use concurrency” The questions are essentially how do we test? how does concurrent programming fit in TDD. Stef
On 6 Sep 2019, at 02:32, Andres Valloud <avalloud@smalltalk.comcastbiz.net> wrote:
TL; DR: generally, developing a good sense of what to do ***before writing a single line of code*** works really well.
On 9/4/19 6:32 , N. Bouraqadi wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org .
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr http://www.synectique.eu / http://www.pharo.org 03 59 35 87 52 Assistant: Julie Jonas FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
On 9/6/19 23:24 , Stéphane Ducasse wrote:
Now it would be nice to read his questions and reply to them and not just tell him “oh do not use concurrency”
I did not say that at all --- yet you are complaining to me :).
The questions are essentially how do we test? how does concurrent programming fit in TDD.
A sharper and more open minded question might be "what is the best way to build reliable programs using concurrency flavor XYZ?". Andres.
Stef
On 6 Sep 2019, at 02:32, Andres Valloud <avalloud@smalltalk.comcastbiz.net <mailto:avalloud@smalltalk.comcastbiz.net>> wrote:
TL; DR: generally, developing a good sense of what to do ***before writing a single line of code*** works really well.
On 9/4/19 6:32 , N. Bouraqadi wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org .
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr http://www.synectique.eu / http://www.pharo.org 03 59 35 87 52 Assistant: Julie Jonas FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
I think the question was if there are any patterns or best practices to identify potential concurrency issues that could be used to build tools... maybe that’s why Complaints occurred...?
Am 07.09.2019 um 10:42 schrieb Andres Valloud <avalloud@smalltalk.comcastbiz.net>:
On 9/6/19 23:24 , Stéphane Ducasse wrote: Now it would be nice to read his questions and reply to them and not just tell him “oh do not use concurrency”
I did not say that at all --- yet you are complaining to me :).
The questions are essentially how do we test? how does concurrent programming fit in TDD.
A sharper and more open minded question might be "what is the best way to build reliable programs using concurrency flavor XYZ?".
Andres.
Stef
On 6 Sep 2019, at 02:32, Andres Valloud <avalloud@smalltalk.comcastbiz.net <mailto:avalloud@smalltalk.comcastbiz.net>> wrote:
TL; DR: generally, developing a good sense of what to do ***before writing a single line of code*** works really well.
On 9/4/19 6:32 , N. Bouraqadi wrote: Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org .
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr http://www.synectique.eu / http://www.pharo.org 03 59 35 87 52 Assistant: Julie Jonas FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
On 7 Sep 2019, at 10:42, Andres Valloud <avalloud@smalltalk.comcastbiz.net> wrote:
On 9/6/19 23:24 , Stéphane Ducasse wrote:
Now it would be nice to read his questions and reply to them and not just tell him “oh do not use concurrency”
I did not say that at all --- yet you are complaining to me :).
Not to you in particular. To everybody :)
The questions are essentially how do we test? how does concurrent programming fit in TDD.
A sharper and more open minded question might be "what is the best way to build reliable programs using concurrency flavor XYZ?”.
No you see you are interpreting the questions of Noury. Again and now this is you: why do you use “sharper” do you imply that the questions of noury were not sharp. :( The point of noury is what is the way to approach concurrency when doing TDD. Now how to build reliable …. Imagine that DrTDD should be extended to support concurrent programming. Then this is the question that we want to get answer. Stef
Stef
On 6 Sep 2019, at 02:32, Andres Valloud <avalloud@smalltalk.comcastbiz.net <mailto:avalloud@smalltalk.comcastbiz.net> <mailto:avalloud@smalltalk.comcastbiz.net <mailto:avalloud@smalltalk.comcastbiz.net>>> wrote:
TL; DR: generally, developing a good sense of what to do ***before writing a single line of code*** works really well.
On 9/4/19 6:32 , N. Bouraqadi wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> <mailto:Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org>> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org> .
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> <mailto:Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org>> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr http://www.synectique.eu / http://www.pharo.org 03 59 35 87 52 Assistant: Julie Jonas FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org>
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org>
Stéphane Ducasse http://stephane.ducasse.free.fr http://www.synectique.eu / http://www.pharo.org 03 59 35 87 52 Assistant: Julie Jonas FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
The point of noury is what is the way to approach concurrency when doing TDD. Now how to build reliable ….
I won’t pretend to answer such a broad question with anything definitive, but I’d start my investigation of concurrency with a look at databases. One of the primary features of any multi-user database (not just GemStone!) is that transactions are in practice concurrent but the system is designed to apply them as if they were serial and in a way where order does not matter. So I would attempt to identify the concurrent tasks and verify that they each got the result that they would have obtained had they been applied in various serial fashions.
Imagine that DrTDD should be extended to support concurrent programming. Then this is the question that we want to get answer.
I would look for a way to pass a set of repeatable tasks to DrTDD and let the testing framework run them in various orders then run them concurrently with interrupts and switching contexts. James
Thanks you all for your answers. Best, Noury
On 7 Sep 2019, at 18:42, James Foster <Smalltalk@JGFoster.net> wrote:
The point of noury is what is the way to approach concurrency when doing TDD. Now how to build reliable ….
I won’t pretend to answer such a broad question with anything definitive, but I’d start my investigation of concurrency with a look at databases. One of the primary features of any multi-user database (not just GemStone!) is that transactions are in practice concurrent but the system is designed to apply them as if they were serial and in a way where order does not matter. So I would attempt to identify the concurrent tasks and verify that they each got the result that they would have obtained had they been applied in various serial fashions.
Imagine that DrTDD should be extended to support concurrent programming. Then this is the question that we want to get answer.
I would look for a way to pass a set of repeatable tasks to DrTDD and let the testing framework run them in various orders then run them concurrently with interrupts and switching contexts.
James _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
Dear Noury, I didn’t had the time to do a thorough web search. You certainly did. I found several references that look very promising. Mny of them were concerning Java and OS threads. Othes are about Erlang (https://mariachris.github.io/Pubs/ERLANG-2011.pdf ). I think not only the concurrency problem but also the base technology matters. I couldn’t find anything about Smalltalk yet. So you are probably conquering new land. Here is a quote from the preface of “Test Driven Development: By Example” from 2003 by Kent Beck: “There certainly are programming tasks that can’t be driven solely by tests (or, at least not yet). Security and concurrency, for example, are two topics where TDD is insufficient to mechanically demonstrate that the goals of the software has been met.” “Subtle concurrency problems can’t be reliably duplicated by running the code.” Good luck! I am looking forward to your findings. Helge Helge Nowak Cincom Smalltalk Technical Account Manager [http://internal.cincom.com/signature/logo.png]<http://www.cincomsmalltalk.com/> Cincom Systems GmbH & Co. oHG Humboldtstraße 3 60318 Frankfurt am Main GERMANY office mobile website email +49 89 89 66 44 94 +49 172 74 00 402 http://www.cincomsmalltalk.com hnowak@cincom.com A standpoint is an intellectual horizon of radius zero. -- Albert Einstein Geschäftsführer/Managing Directors: Thomas M. Nies, Donald E. Vick oHG mit Sitz/based in Frankfurt am Main (Amtsgericht Frankfurt am Main HRA 50297) Pers. haftender Gesellschafter/Partner liable to unlimited extent: Cincom Systems Verwaltungsgesellschaft mbH (Amtsgericht Königstein/Ts. HRB 5069) --- CONFIDENTIALITY STATEMENT --- This e-mail transmission contains information that is intended to be privileged and confidential. It is intended only for the addressee named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited, please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. From: Esug-list <esug-list-bounces@lists.esug.org> On Behalf Of Noury Bouraqadi Sent: Tuesday, 10 September 2019 15:12 To: Members ESUG <esug-list@lists.esug.org> Subject: Re: [Esug-list] [SPAM] Re: Concurrency Best Practices + Tests Thanks you all for your answers. Best, Noury On 7 Sep 2019, at 18:42, James Foster <Smalltalk@JGFoster.net<mailto:Smalltalk@JGFoster.net>> wrote: The point of noury is what is the way to approach concurrency when doing TDD. Now how to build reliable …. I won’t pretend to answer such a broad question with anything definitive, but I’d start my investigation of concurrency with a look at databases. One of the primary features of any multi-user database (not just GemStone!) is that transactions are in practice concurrent but the system is designed to apply them as if they were serial and in a way where order does not matter. So I would attempt to identify the concurrent tasks and verify that they each got the result that they would have obtained had they been applied in various serial fashions. Imagine that DrTDD should be extended to support concurrent programming. Then this is the question that we want to get answer. I would look for a way to pass a set of repeatable tasks to DrTDD and let the testing framework run them in various orders then run them concurrently with interrupts and switching contexts. James _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org<mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
Hi Noury, I didn't respond because I really didn't think I had much to add but I've been thinking about what I would want to have in a concurrent system. This may be obvious but Concurrent systems promise either speed or scalability. The goal will change the requirements. We know that for speed there are two aspects to consider. You can only go as fast as it takes for your largest component to process and limited resources means that going really fast on easy to process components of the problem set will only lead to a large backlog and coordination issues. For scalability you need to ensure completeness. You can't just send stuff off and assume it will get processed. There is a lot to do to ensure that the everything that fails or is delayed is rescheduled to prevent a major backup. Both of these goals, it would seem to me, would be helped considerably by a system that can explain itself. Depending on the size of the processing task this could get tricky. Something that processes millions of messages a second would need to have a much more sophisticated system to gather information without changing the efficiency of the system itself. Something that is less time intensive could process and keep this information with the processed results. What I would want to know is time to process all of the components compared to time to process the full solution. I would also want to know about errors, rescheduling, wait times. Ultimately the best concurrent system will run at time to run slowest component + coordination time. If the system is not running at that speed we would like to have information as to why. Is there a backup, deadlock or other errors, resource issues or something else that is causing the system to delay. Comparing the processing time of the full process and the processing time of each component could tell you a lot about where things are going wrong (maybe your process tree is wrong, or your scheduling is too aggressive causing rescheduling on a regular basis or 2 times the processing time of a component because expected data wasn't available). If this is a huge system then you can't really keep all that information but you could keep averages instead. The idea is probably not that novel but having a system that could explain itself, what it is doing, and how well it is performing, could be a very good debugging tool. I would be very careful to build it such that it doesn't change the processing efficiency of the production system and that the explanation system is optional (I know they don't really go together [optional and no effect] which makes this very difficult). That also means that I wouldn't try to use this reporting system to automatically change the system itself. Hope that helps. All the best, *Ron Teitelbaum* *Chief Executive Officer**3D Immersive Collaboration Consulting, LLC* ron@3dicc.com www.3dicc.com <https://www.facebook.com/3DICC> <https://twitter.com/RonTeitelbaum> <https://www.linkedin.com/in/ronteitelbaum> On Tue, Sep 10, 2019 at 9:52 AM Nowak, Helge <HNowak@cincom.com> wrote:
Dear Noury,
I didn’t had the time to do a thorough web search. You certainly did. I found several references that look very promising. Mny of them were concerning Java and OS threads. Othes are about Erlang ( https://mariachris.github.io/Pubs/ERLANG-2011.pdf ). I think not only the concurrency problem but also the base technology matters. I couldn’t find anything about Smalltalk yet. So you are probably conquering new land.
Here is a quote from the preface of “Test Driven Development: By Example” from 2003 by Kent Beck: “There certainly are programming tasks that can’t be driven solely by tests (or, at least not yet). Security and concurrency, for example, are two topics where TDD is insufficient to mechanically demonstrate that the goals of the software has been met.” “Subtle concurrency problems can’t be reliably duplicated by running the code.”
Good luck! I am looking forward to your findings.
Helge
*Helge Nowak *Cincom Smalltalk Technical Account Manager
[image: http://internal.cincom.com/signature/logo.png] <http://www.cincomsmalltalk.com/> Cincom Systems GmbH & Co. oHG Humboldtstraße 3 60318 Frankfurt am Main GERMANY
office mobile
website email
+49 89 89 66 44 94 +49 172 74 00 402
http://www.cincomsmalltalk.com hnowak@cincom.com
*A standpoint is an intellectual horizon of radius zero. -- Albert Einstein *
Geschäftsführer/Managing Directors: Thomas M. Nies, Donald E. Vick oHG mit Sitz/based in Frankfurt am Main (Amtsgericht Frankfurt am Main HRA 50297) Pers. haftender Gesellschafter/Partner liable to unlimited extent: Cincom Systems Verwaltungsgesellschaft mbH (Amtsgericht Königstein/Ts. HRB 5069)
--- CONFIDENTIALITY STATEMENT ---
This e-mail transmission contains information that is intended to be privileged and confidential. It is intended only for the addressee named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited, please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.
*From:* Esug-list <esug-list-bounces@lists.esug.org> *On Behalf Of *Noury Bouraqadi *Sent:* Tuesday, 10 September 2019 15:12 *To:* Members ESUG <esug-list@lists.esug.org> *Subject:* Re: [Esug-list] [SPAM] Re: Concurrency Best Practices + Tests
Thanks you all for your answers.
Best,
Noury
On 7 Sep 2019, at 18:42, James Foster <Smalltalk@JGFoster.net> wrote:
The point of noury is what is the way to approach concurrency when doing TDD.
Now how to build reliable ….
I won’t pretend to answer such a broad question with anything definitive, but I’d start my investigation of concurrency with a look at databases. One of the primary features of any multi-user database (not just GemStone!) is that transactions are in practice concurrent but the system is designed to apply them as if they were serial and in a way where order does not matter. So I would attempt to identify the concurrent tasks and verify that they each got the result that they would have obtained had they been applied in various serial fashions.
Imagine that DrTDD should be extended to support concurrent programming.
Then this is the question that we want to get answer.
I would look for a way to pass a set of repeatable tasks to DrTDD and let the testing framework run them in various orders then run them concurrently with interrupts and switching contexts.
James
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
Hey Ron, Part of what you’ve expressed has been distilled into a ‘law’. Amdahl’s law describes the theoretical speedup in execution latency. https://en.wikipedia.org/wiki/Amdahl%27s_law <https://en.wikipedia.org/wiki/Amdahl's_law> Note, concurrency alone would not offer this speedup. This requires parallel execution. Without parallelism, you would still need to wait the aggregate computation time for all concurrent components. The latency of the longest computation is not enough. Kurt
On Sep 10, 2019, at 9:41 AM, Ron Teitelbaum <ron@3dicc.com> wrote:
Hi Noury,
I didn't respond because I really didn't think I had much to add but I've been thinking about what I would want to have in a concurrent system. This may be obvious but
Concurrent systems promise either speed or scalability. The goal will change the requirements. We know that for speed there are two aspects to consider. You can only go as fast as it takes for your largest component to process and limited resources means that going really fast on easy to process components of the problem set will only lead to a large backlog and coordination issues.
For scalability you need to ensure completeness. You can't just send stuff off and assume it will get processed. There is a lot to do to ensure that the everything that fails or is delayed is rescheduled to prevent a major backup.
Both of these goals, it would seem to me, would be helped considerably by a system that can explain itself. Depending on the size of the processing task this could get tricky. Something that processes millions of messages a second would need to have a much more sophisticated system to gather information without changing the efficiency of the system itself. Something that is less time intensive could process and keep this information with the processed results.
What I would want to know is time to process all of the components compared to time to process the full solution. I would also want to know about errors, rescheduling, wait times. Ultimately the best concurrent system will run at time to run slowest component + coordination time. If the system is not running at that speed we would like to have information as to why. Is there a backup, deadlock or other errors, resource issues or something else that is causing the system to delay. Comparing the processing time of the full process and the processing time of each component could tell you a lot about where things are going wrong (maybe your process tree is wrong, or your scheduling is too aggressive causing rescheduling on a regular basis or 2 times the processing time of a component because expected data wasn't available). If this is a huge system then you can't really keep all that information but you could keep averages instead.
The idea is probably not that novel but having a system that could explain itself, what it is doing, and how well it is performing, could be a very good debugging tool. I would be very careful to build it such that it doesn't change the processing efficiency of the production system and that the explanation system is optional (I know they don't really go together [optional and no effect] which makes this very difficult). That also means that I wouldn't try to use this reporting system to automatically change the system itself.
Hope that helps. All the best,
Ron Teitelbaum Chief Executive Officer 3D Immersive Collaboration Consulting, LLC ron@3dicc.com <mailto:ron@3dicc.com> www.3dicc.com <http://www.3dicc.com/> <https://www.facebook.com/3DICC> <https://twitter.com/RonTeitelbaum> <https://www.linkedin.com/in/ronteitelbaum>
On Tue, Sep 10, 2019 at 9:52 AM Nowak, Helge <HNowak@cincom.com <mailto:HNowak@cincom.com>> wrote: Dear Noury,
I didn’t had the time to do a thorough web search. You certainly did. I found several references that look very promising. Mny of them were concerning Java and OS threads. Othes are about Erlang (https://mariachris.github.io/Pubs/ERLANG-2011.pdf <https://mariachris.github.io/Pubs/ERLANG-2011.pdf> ). I think not only the concurrency problem but also the base technology matters. I couldn’t find anything about Smalltalk yet. So you are probably conquering new land.
Here is a quote from the preface of “Test Driven Development: By Example” from 2003 by Kent Beck: “There certainly are programming tasks that can’t be driven solely by tests (or, at least not yet). Security and concurrency, for example, are two topics where TDD is insufficient to mechanically demonstrate that the goals of the software has been met.” “Subtle concurrency problems can’t be reliably duplicated by running the code.”
Good luck! I am looking forward to your findings.
Helge
Helge Nowak Cincom Smalltalk Technical Account Manager
<image001.png> <http://www.cincomsmalltalk.com/> Cincom Systems GmbH & Co. oHG Humboldtstraße 3 60318 Frankfurt am Main GERMANY
office mobile
website email
+49 89 89 66 44 94 +49 172 74 00 402
http://www.cincomsmalltalk.com <http://www.cincomsmalltalk.com/> hnowak@cincom.com <mailto:hnowak@cincom.com> A standpoint is an intellectual horizon of radius zero. -- Albert Einstein
Geschäftsführer/Managing Directors: Thomas M. Nies, Donald E. Vick oHG mit Sitz/based in Frankfurt am Main (Amtsgericht Frankfurt am Main HRA 50297) Pers. haftender Gesellschafter/Partner liable to unlimited extent: Cincom Systems Verwaltungsgesellschaft mbH (Amtsgericht Königstein/Ts. HRB 5069)
--- CONFIDENTIALITY STATEMENT ---
This e-mail transmission contains information that is intended to be privileged and confidential. It is intended only for the addressee named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited, please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.
From: Esug-list <esug-list-bounces@lists.esug.org <mailto:esug-list-bounces@lists.esug.org>> On Behalf Of Noury Bouraqadi Sent: Tuesday, 10 September 2019 15:12 To: Members ESUG <esug-list@lists.esug.org <mailto:esug-list@lists.esug.org>> Subject: Re: [Esug-list] [SPAM] Re: Concurrency Best Practices + Tests
Thanks you all for your answers.
Best,
Noury
On 7 Sep 2019, at 18:42, James Foster <Smalltalk@JGFoster.net <mailto:Smalltalk@JGFoster.net>> wrote:
The point of noury is what is the way to approach concurrency when doing TDD.
Now how to build reliable ….
I won’t pretend to answer such a broad question with anything definitive, but I’d start my investigation of concurrency with a look at databases. One of the primary features of any multi-user database (not just GemStone!) is that transactions are in practice concurrent but the system is designed to apply them as if they were serial and in a way where order does not matter. So I would attempt to identify the concurrent tasks and verify that they each got the result that they would have obtained had they been applied in various serial fashions.
Imagine that DrTDD should be extended to support concurrent programming.
Then this is the question that we want to get answer.
I would look for a way to pass a set of repeatable tasks to DrTDD and let the testing framework run them in various orders then run them concurrently with interrupts and switching contexts.
James
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org>
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org <mailto:Esug-list@lists.esug.org> http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org <http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org> _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
Hi Kurt, Thank you for that link. I'd assumed parallel computing as the reason for concurrency. Maybe that was a mistake? I called the individual parallel processes components of the problem. I also think in terms of the dependency tree. You really can't get any faster than the the longest branch which resolves all the dependents processes required to get a a result even if those dependents result from parallel processing. Also in practice it doesn't make sense to only consider the processing time of a component since there is overhead in coordination and scheduling. The maximum possible speed up would approach Amdahl’s law as the overhead time approaches 0 (which I guess is the point of a theoretical maximum). If on the other hand we are talking about handling multiple problems sets concurrently and not parallel processing there are other issues to consider. Mostly the problem becomes one of identifying available resources and optimizing queuing processes. It doesn't make sense to have a tiny process wait behind one that will take a very long time. Still you need to queue based on something. In that case I would spend more time on building in process time estimates and handling dynamic rescheduling based on time to process vs current age in queue. The reporting system (a system that explained what it was doing) would still work to help debug the system but you would be looking to optimize the schedule flow to improve efficiency. Collecting data about how it was scheduled (total work in queue at schedule, at processing, after processing), how long it waited, the difference between estimated processing time and actual processing time would be more useful. Noury, did you ever share details of the actual problem you are trying to solve? All the best, *Ron Teitelbaum* *Chief Executive Officer**3D Immersive Collaboration Consulting, LLC* ron@3dicc.com www.3dicc.com <https://www.facebook.com/3DICC> <https://twitter.com/RonTeitelbaum> <https://www.linkedin.com/in/ronteitelbaum> On Tue, Sep 10, 2019 at 1:24 PM Kurt Kilpela <self@kurtkilpela.com> wrote:
Hey Ron,
Part of what you’ve expressed has been distilled into a ‘law’. Amdahl’s law describes the theoretical speedup in execution latency.
https://en.wikipedia.org/wiki/Amdahl%27s_law <https://en.wikipedia.org/wiki/Amdahl's_law>
Note, concurrency alone would not offer this speedup. This requires parallel execution. Without parallelism, you would still need to wait the aggregate computation time for all concurrent components. The latency of the longest computation is not enough.
Kurt
On Sep 10, 2019, at 9:41 AM, Ron Teitelbaum <ron@3dicc.com> wrote:
Hi Noury,
I didn't respond because I really didn't think I had much to add but I've been thinking about what I would want to have in a concurrent system. This may be obvious but
Concurrent systems promise either speed or scalability. The goal will change the requirements. We know that for speed there are two aspects to consider. You can only go as fast as it takes for your largest component to process and limited resources means that going really fast on easy to process components of the problem set will only lead to a large backlog and coordination issues.
For scalability you need to ensure completeness. You can't just send stuff off and assume it will get processed. There is a lot to do to ensure that the everything that fails or is delayed is rescheduled to prevent a major backup.
Both of these goals, it would seem to me, would be helped considerably by a system that can explain itself. Depending on the size of the processing task this could get tricky. Something that processes millions of messages a second would need to have a much more sophisticated system to gather information without changing the efficiency of the system itself. Something that is less time intensive could process and keep this information with the processed results.
What I would want to know is time to process all of the components compared to time to process the full solution. I would also want to know about errors, rescheduling, wait times. Ultimately the best concurrent system will run at time to run slowest component + coordination time. If the system is not running at that speed we would like to have information as to why. Is there a backup, deadlock or other errors, resource issues or something else that is causing the system to delay. Comparing the processing time of the full process and the processing time of each component could tell you a lot about where things are going wrong (maybe your process tree is wrong, or your scheduling is too aggressive causing rescheduling on a regular basis or 2 times the processing time of a component because expected data wasn't available). If this is a huge system then you can't really keep all that information but you could keep averages instead.
The idea is probably not that novel but having a system that could explain itself, what it is doing, and how well it is performing, could be a very good debugging tool. I would be very careful to build it such that it doesn't change the processing efficiency of the production system and that the explanation system is optional (I know they don't really go together [optional and no effect] which makes this very difficult). That also means that I wouldn't try to use this reporting system to automatically change the system itself.
Hope that helps.
All the best,
*Ron Teitelbaum* *Chief Executive Officer**3D Immersive Collaboration Consulting, LLC* ron@3dicc.com www.3dicc.com
<https://www.facebook.com/3DICC> <https://twitter.com/RonTeitelbaum> <https://www.linkedin.com/in/ronteitelbaum>
On Tue, Sep 10, 2019 at 9:52 AM Nowak, Helge <HNowak@cincom.com> wrote:
Dear Noury,
I didn’t had the time to do a thorough web search. You certainly did. I found several references that look very promising. Mny of them were concerning Java and OS threads. Othes are about Erlang ( https://mariachris.github.io/Pubs/ERLANG-2011.pdf ). I think not only the concurrency problem but also the base technology matters. I couldn’t find anything about Smalltalk yet. So you are probably conquering new land.
Here is a quote from the preface of “Test Driven Development: By Example” from 2003 by Kent Beck: “There certainly are programming tasks that can’t be driven solely by tests (or, at least not yet). Security and concurrency, for example, are two topics where TDD is insufficient to mechanically demonstrate that the goals of the software has been met.” “Subtle concurrency problems can’t be reliably duplicated by running the code.”
Good luck! I am looking forward to your findings.
Helge
*Helge Nowak *Cincom Smalltalk Technical Account Manager
<image001.png> <http://www.cincomsmalltalk.com/> Cincom Systems GmbH & Co. oHG Humboldtstraße 3 60318 Frankfurt am Main GERMANY
office mobile
website email
+49 89 89 66 44 94 +49 172 74 00 402
http://www.cincomsmalltalk.com hnowak@cincom.com
*A standpoint is an intellectual horizon of radius zero. -- Albert Einstein *
Geschäftsführer/Managing Directors: Thomas M. Nies, Donald E. Vick oHG mit Sitz/based in Frankfurt am Main (Amtsgericht Frankfurt am Main HRA 50297) Pers. haftender Gesellschafter/Partner liable to unlimited extent: Cincom Systems Verwaltungsgesellschaft mbH (Amtsgericht Königstein/Ts. HRB 5069)
--- CONFIDENTIALITY STATEMENT ---
This e-mail transmission contains information that is intended to be privileged and confidential. It is intended only for the addressee named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited, please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.
*From:* Esug-list <esug-list-bounces@lists.esug.org> *On Behalf Of *Noury Bouraqadi *Sent:* Tuesday, 10 September 2019 15:12 *To:* Members ESUG <esug-list@lists.esug.org> *Subject:* Re: [Esug-list] [SPAM] Re: Concurrency Best Practices + Tests
Thanks you all for your answers.
Best,
Noury
On 7 Sep 2019, at 18:42, James Foster <Smalltalk@JGFoster.net> wrote:
The point of noury is what is the way to approach concurrency when doing TDD.
Now how to build reliable ….
I won’t pretend to answer such a broad question with anything definitive, but I’d start my investigation of concurrency with a look at databases. One of the primary features of any multi-user database (not just GemStone!) is that transactions are in practice concurrent but the system is designed to apply them as if they were serial and in a way where order does not matter. So I would attempt to identify the concurrent tasks and verify that they each got the result that they would have obtained had they been applied in various serial fashions.
Imagine that DrTDD should be extended to support concurrent programming.
Then this is the question that we want to get answer.
I would look for a way to pass a set of repeatable tasks to DrTDD and let the testing framework run them in various orders then run them concurrently with interrupts and switching contexts.
James
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
On 9/7/19 9:03 , Stéphane Ducasse wrote:
Again and now this is you: why do you use “sharper” do you imply that the questions of noury were not sharp. :(
No, I didn't say that either --- "sharper", not "sharp vs dull".
The point of noury is what is the way to approach concurrency when doing TDD. Now how to build reliable ….
Imagine that DrTDD should be extended to support concurrent programming. Then this is the question that we want to get answer.
To me, the interesting part is how to build reliable software, so I'd prioritize that over choice of tools. Because who knows, maybe you look at reliability and realize instead of TDD you need XYZ. Andres.
On 9/7/19 9:03 , Stéphane Ducasse wrote:
Again and now this is you: why do you use “sharper” do you imply that the questions of noury were not sharp. :(
No, I didn't say that either --- "sharper", not "sharp vs dull”.
At least in french, when we say that "a better/sharper question would be" it implies that previous question was not sharp/smart. May be in english this is not the case, I’m sorry my english is rather basic. I thought that adding “er” at the end of adjective meant more. So by construction it means that other are less. I do not know I’m not good in logic either. May be you do see it but I felt that you were saying that and this is why I reacted because I do not like when people lower other people in this mailing-list. Of course I could have say using the same pattern, that a nicer/smarter/shaper answer would have been not to show off and to reply to the question. And when another person would have mentioned it, I would have decline my intention.
The point of noury is what is the way to approach concurrency when doing TDD. Now how to build reliable ….
Imagine that DrTDD should be extended to support concurrent programming. Then this is the question that we want to get answer.
To me, the interesting part is how to build reliable software, so I'd prioritize that over choice of tools. Because who knows, maybe you look at reliability and realize instead of TDD you need XYZ.
Again, as professional developers and teachers, I believe that we know all that. But thanks for your wisdom. It seemed to me that it was quite a common and basic knowledge. But again this was not the questions. Stef
Andres. _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr http://www.synectique.eu / http://www.pharo.org 03 59 35 87 52 Assistant: Julie Jonas FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
Some random thoughts: 1) model your concurrency problem before you start with TDD, e.g. using TLA+. This will make clear what you need to implement using TDD. 2) once you have a satisfying result from 1, you can start designing your solution using TDD. Of the tests that you will need to write: - some will be pure behaviour tests which don’t to worry about concurrency - some will be coordination tests were you need to worry about concurrency. The coordination tests can be approached as simulation problems, where you can manage the choreography of the various concurrent processes. To do this: A) model your concurrent operation as transactions. Split the transaction execution into smaller steps, that can be tested separately. You can then set up a sequence of operations coming from multiple processes, interleaving ops from one process with those from the others. This will also make it easier to understand where to use coordination primitives. B) Explicitly manage the evaluation of the code in the two processes. You can do this with a creative use of method wrappers, Delay objects etc., or by forcing the code to run through a debugger, so that can manage the order of the message sends between the processes Please also note that TDD might not be enough and you might need to run other type of tests than TDD-style unit tests. Cheers, Giovanni Il giorno sab 7 set 2019 alle 03:25 Stéphane Ducasse < stephane.ducasse@inria.fr> ha scritto:
Hi guys
I imagine that noury knows most of the remarks you sent it. He is far from dull. Now it would be nice to read his questions and reply to them and not just tell him “oh do not use concurrency”
The questions are essentially how do we test? how does concurrent programming fit in TDD.
Stef
On 6 Sep 2019, at 02:32, Andres Valloud <avalloud@smalltalk.comcastbiz.net> wrote:
TL; DR: generally, developing a good sense of what to do ***before writing a single line of code*** works really well.
On 9/4/19 6:32 , N. Bouraqadi wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Thanks for your help, Noury _______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org .
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr http://www.synectique.eu / http://www.pharo.org 03 59 35 87 52 Assistant: Julie Jonas FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley <https://www.google.com/maps/search/40,+avenue+Halley?entry=gmail&source=g> , Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
participants (14)
-
Andres Valloud -
Giovanni Corriga -
James Foster -
Joachim Tuchel -
John M McIntosh -
Karsten Kusche -
Kurt Kilpela -
Maarten Mostert -
N. Bouraqadi -
Noury Bouraqadi
-
Nowak, Helge -
Ron Teitelbaum -
Stéphane Ducasse -
Tim Mackinnon