Current Events > Let's talk software for scientific computing and plotting

(+) Yeah! (+)

Hi. I've been working now for two years in MATLAB, and I'm getting more and more curious to broaden my horizons. So I figured we could have a nice exchange on experiences with different software and languages.

I think it's still pretty well established that FORTRAN is king if you have tons of really heavy stuff to compute. Personally I'm not at that level so I'm looking elsewhere.

- MATLAB is actually really convenient. Sometimes I really miss the extra muscle you get from a "proper" programming language. Like, it's a little annoying that for loops are actually slower than vectorized operations, and it irks me that I need to store arrays in variables before I can index into them. It also annoys me a little how debugging a function differs from debugging a script, etc. The plotting is mostly really intuitive, but it always looks so... MATLABy. It feels really corny to put MATLAB plus into papers.
- I have been fiddling around a little with Python and NumPy, and I like it plenty. Python is one of my favorite languages, but I actually prefer it for problems that are a bit more general purpose/computer sciencey. Having a workspace with a nice GUI is really convenient, and Python doesn't really come with such batteries included. Matplotlib is fine, but I don't think it necessarily looks better than MATLAB.

What I'm really curious about is Julia. It seems to strike the sweet spot between productive expressiveness and high performance. The plotting also looks pretty cool. Anyone tried it?
I'm also curious about R and GNUplot for plotting purposes, but those don't go well together with MATLAB, given that it's not great for output redirection etc.

@Darkman124 @COVxy (You guys are the only ones I know on here who would have relevant experience)
---

I primarily use MATLAB, though some of my newer work is in simulation and theoretical neuroscience, so I've been using Python for that.

I have to say, over the years, I've gotten a real appreciation for "thinking in matrices". The biggest hurdle for me to switch entirely over to python is it's clunky syntax for matrix operations.

As far as plotting, I think MATLAB can make absolutely beautiful plots if you spend enough time creating custom functions so that you're not constantly recreating all the aesthetics.


This is just built off of the default kernel density plot (ksdensity), and it seems to look quite nice to me.
---

I definitely agree with the clunky matrix syntax in Python. It's one of the reasons I've yet to code up comparisons for some of my routines in NumPy.

I will admit, when I say that MATLAB is boring for plotting, I'm being a bit disingenuous - I've almost exclusively been using plot() with the default color scheme. I did use contourf() recently to make custom spectrograms, and it definitely isn't as bad as I make it out to be.
However, for a revision I did for a paper recently I had to add a fourth subplot to figures with three subplots. That was really unnecessarily complicated. And the Julia capabilities to make LaTeX plots looks attractive. I guess MATLAB has that, too, but like I said it's annoying to be shut into a sandbox rather than being able to pipe data from wherever into plotting scripts in bash.
---

I will agree that MATLAB's subplot function sucks. Absolutely awful timesuck with certain amounts of guaranteed failure.

I almost exclusively plot things separately, save them, and then put them together quickly in GIMP. The only time I've forced myself to really fuck with subplot was when I was generating a quality control pipeline for the lab, so that when we collect data we can just feed it in and get a giant sheet of QC plots from the data.
---

btw, I've also been curious about Julia, it was hinted at a couple of years ago at the end of a scientific computing class I took. Never got the chance to try it out though.
---

I tried to get into ggplot2 a couple of years ago and found the geom shit too counter-intuitive to be useful, so I gave up =P.
---

Tinkering around with subplot has been really useful for me when I've had to make a large number of plots. So when I completely automate the plotting and you can get all the data you want in one figure, it's super convenient. So, similar to your case I guess. Doing it by hand, though, and editing them, sucks big time.

About Julia, I'm hoping to get some time over soon and reimplement some of my routines. However some of them rely on splining data that's not uniformly sampled, which I can't find good support for. I think I can prep it first in MATLAB though and just start off by upsampling it once. That's a really attractive feature of Julia - being able to run entire MATLAB/R/Python environments seamlessly in addition to being able to call C and FORTRAN routines. And since it can both read and write .mat files without problems, it'll be easy to hook in individual modules into my processing chain.
---

python's ability to call codes from other software packages is amazing

a structural engineer and i worked together to create a coupled script that deformed propellant as it burned and created high-pressure hot gasses, then called up a little script I wrote to calculate the ideal gas law dynamics.

i can't help you with julia though sorry
---

If you use R, I would recommend RStudio as an interface for R.
---

I'm not nearly as hardcore into this as most of you, but I agree that python is amazing when it comes to working with other software. I have only a tiny amount of experience with MATLAB, but I suspect when it comes to dedicated mathy stuff it will have a lot of the tools you need already built-in and readily available.
---

If you would like the option for on the fly analysis and plotting in Python, you can check out Spyder, very similar to MATLAB's interfacing. Also integrates well with Conda.
---

I'll look Into that, thanks!
And yeah, Python's strength imo is it's general purpose capabilities. For example I could easily write a web crawler to get whatever data I need then just call another process or pipe the output there or whatever, and it'll take me like no time at all, not to mention the insanely convenient IO. A NetCDF file just becomes a native object and I don't have to worry about using read functions or anything. It's just overwhelmingly straightforward sometimes.
---

Bonus question: Does anyone do any GPGPU calculations? In my field the general consensus is that a GPU would offer uncertain gains that wouldn't be worth the cost, at least for people on the operational side of things. That makes sense, why spend a lot of money on putting your processing on a GPU if you're not sure you'll get a speedup and processing time of a week isn't so bad?
But it still feels like there should be a larger use for it than there is.
---

Stupid edit button next to the delete button -_-

I've used GNUplot. I like it. It's simple to use, and it's latex compatibility is a plus for me. It's my plotter of choice, certainly more so than excel.

Until now I've only used C++ for mathematical programming, but my micro process engineering course next semester will include MATLAB tutorials. It'll be interesting to see how that goes.
---

I'm a big Python fan. I haven't used Matlab in a long time, but it seems like numpy and matplotlib give you basically the same functionality and similar syntax as vanilla Matlab. I think Matlab is supposed to have some nice proprietary libraries for some fields, but obviously Python also has a large ecosystem of scientific and data analysis libraries.

For a more comprehensive IDE experience, try Spyder or Enthought Canopy. I also highly recommend using either the Anaconda or Enthought package managers.

For speeding up computation, especially loops, I've had success using the numba package, which performs just-in-time compilation of functions as long as they don't stray too far from basic numpy and Python.
---

Yeah, a big pro with MATLAB is that it comes with batteries included (as long as you pay for it). Tons of great signal processing, machine learning, statistical stuff, symbolic, etc etc. No doubt you can find equivalent stuff for most of it in Python, but you'll have to find it, and trust that it's good enough, etc.
---

Python > MATLAB hands down.
---

C++, man
---

MutantJohn posted...

C++, man

I think my life would be a whole lot more unnecessarily difficult if I did all my data analysis in C++.
---

COVxy posted...

MutantJohn posted...

C++, man

I think my life would be a whole lot more unnecessarily difficult if I did all my data analysis in C++.

How much C++ do you know?
---

MutantJohn posted...

How much C++ do you know?

Enough to know that there's almost no reason to deal with the low level garbage handling that you need to do in C++ for almost all data analysis.
---

COVxy posted...

MutantJohn posted...

How much C++ do you know?

Enough to know that there's almost no reason to deal with the low level garbage handling that you need to do in C++ for almost all data analysis.

Ah, basically not enough to be productive with it yet. Start learning it for real.
---

MutantJohn posted...

Ah, basically not enough to be productive with it yet. Start learning it for real.

I really don't see why it would be an appropriate language. I mean, if you need to code underlying software to do on the fly analysis for some hardware or something, I could see. But 99% of scientific computing does not deal with that.
(and to my knowledge, people typically use FORTRAN for that, like TC noted)
---

Jupyter Notebook is crazy good. It's a web application where you can write both text (markup language) and code (such as python or R) in a presentable way. Output generated by the code such as tables and graphs gets displayed in the notebook.

https://try.jupyter.org/
---

COVxy posted...

MutantJohn posted...

Ah, basically not enough to be productive with it yet. Start learning it for real.

I really don't see why it would be an appropriate language. I mean, if you need to code underlying software to do on the fly analysis for some hardware or something, I could see. But 99% of scientific computing does not deal with that.
(and to my knowledge, people typically use FORTRAN for that, like TC noted)

I was talking to a PhD student who needed 2 weeks of wall time to run her R code. I lol'd. Most scientists aren't good enough at programming to use C++ effectively but most all real code is backed by C++ and if you're using old libs, C and Fortran.
---

Personally, I'd recommend Julia.
---

FightingGames posted...

Jupyter Notebook is crazy good. It's a web application where you can write both text (markup language) and code (such as python or R) in a presentable way. Output generated by the code such as tables and graphs gets displayed in the notebook.

https://try.jupyter.org/

Yeah I like jupyter too it's a nice thing.MutantJohn posted...

COVxy posted...

MutantJohn posted...

How much C++ do you know?

Enough to know that there's almost no reason to deal with the low level garbage handling that you need to do in C++ for almost all data analysis.

Ah, basically not enough to be productive with it yet. Start learning it for real.

Damn
---

MutantJohn posted...

I was talking to a PhD student who needed 2 weeks of wall time to run her R code. I lol'd. Most scientists aren't good enough at programming to use C++ effectively but most all real code is backed by C++ and if you're using old libs, C and Fortran.

I mean, the increased processing time (which definitely is true, given these higher level languages) is probably worth the time saved code writing time for each analysis.

There's a very pragmatic reason why these higher level languages have become popular for scientific computing. There is a problem with over optimization.
---

COVxy posted...

MutantJohn posted...

I was talking to a PhD student who needed 2 weeks of wall time to run her R code. I lol'd. Most scientists aren't good enough at programming to use C++ effectively but most all real code is backed by C++ and if you're using old libs, C and Fortran.

I mean, the increased processing time (which definitely is true, given these higher level languages) is probably worth the time saved code writing time for each analysis.

There's a very pragmatic reason why these higher level languages have become popular for scientific computing. There is a problem with over optimization.

It's not over-optimization. C++ just takes a long time to learn and a lot if discipline to use. Most scientists lack the required passion for programming to learn it effectively.
---

MutantJohn posted...

It's not over-optimization. C++ just takes a long time to learn and a lot if discipline to use. Most scientists lack the required passion for programming to learn it effectively.

It is, there's no reason to work in such a complex language (as in low level, really) to perform a linear regression. The time benefit is minuscule in processing time, but the coding time is very large. This trade off, for most problems, holds pretty true.
---

COVxy posted...

It is, there's no reason to work in such a complex language to perform a linear regression. The time benefit is minuscule in processing time, but the coding time is very large. This trade off, for most problems, holds pretty true.

^ this. No need to reinvent the wheel unless you need a special custom-built wheel of some sort, lol.
---

MutantJohn posted...

COVxy posted...

MutantJohn posted...

I was talking to a PhD student who needed 2 weeks of wall time to run her R code. I lol'd. Most scientists aren't good enough at programming to use C++ effectively but most all real code is backed by C++ and if you're using old libs, C and Fortran.

I mean, the increased processing time (which definitely is true, given these higher level languages) is probably worth the time saved code writing time for each analysis.

There's a very pragmatic reason why these higher level languages have become popular for scientific computing. There is a problem with over optimization.

It's not over-optimization. C++ just takes a long time to learn and a lot if discipline to use. Most scientists lack the required passion for programming to learn it effectively.

Word. To be fair though, it is a better use of time to use something like Python for your field than waste time learning a real language for marginal benefits.

Python has a lot of wrappers too, opencv is in Python but uses a C backbone. Custom work needs C for sure though.
---

MutantJohn posted...

COVxy posted...

MutantJohn posted...

Ah, basically not enough to be productive with it yet. Start learning it for real.

I really don't see why it would be an appropriate language. I mean, if you need to code underlying software to do on the fly analysis for some hardware or something, I could see. But 99% of scientific computing does not deal with that.
(and to my knowledge, people typically use FORTRAN for that, like TC noted)

I was talking to a PhD student who needed 2 weeks of wall time to run her R code. I lol'd. Most scientists aren't good enough at programming to use C++ effectively but most all real code is backed by C++ and if you're using old libs, C and Fortran.

Most scientists aren't using C++ because the longer execution time is greatly offset by the time saved from a much more expressive language.
My background is game technology and software engineering, I'm well aware that C++ can outperform something like R or MATLAB. I'm also well aware that if I'd been working in C++ instead of MATLAB for these last two years, I wouldn't have gotten half as far as I have now.
And you're a bit off on your comment about only old libs using C and FORTRAN. A lot of huge mainframe code is still FORTRAN, which motivates an up to date ecosystem of tools and libs. And a lot of embedded systems will typically run C on some Linux machine, so C++ isn't the clear replacement people thought it would be.
---

This is why I'm happy people like me exist. I'm here to make your shit fast.
---

2 weeks of time for someone to run their code? that's crazy. I wonder if they could have taken advantage of some kind of parallel programming to speed that up
---

MutantJohn posted...

C++ just takes a long time to learn and a lot if discipline to use. Most scientists lack the required passion for programming to learn it effectively.

Writing C++ also takes longer. If it didn't, there would be literally no benefit to use high-level scripting languages. And when the biggest time cost is processing massive amounts of data with routines that were already optimized in C or FORTRAN, you're not going to want to double it triple the time you spend programming to shave off some 10% of overhead on a script you'll be running for a month then never touch again.
---

Sativa_Rose posted...

2 weeks of time for someone to run their code? that's crazy. I wonder if they could have taken advantage of some kind of parallel programming to speed that up

It's not uncommon. A colleague of mine does research that involves building huge graphs on data from social networks (who likes what etc) and traversing them. He can distribute that on many machines as much as he wants, things are still going to take time.
---