About memory usage with Fiji/JVM

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

About memory usage with Fiji/JVM

Hadrien Mary
Hi,

New SCIFIO I/O system in Fiji/IJ2 automatically detects wether virtual
stack is needed according to system memory available and image size
(if I understood correctly).

Virtual stack is fine but it always better to work without when you
can. For example working with timelapse stacks, sliding time is much
faster when image is not a virtual stack (and it makes sense since RAM
memory access is much faster than hard drive memory access...).

Saying that I noticed something: when I close a big image (no virtual
stack because moving time slider is fast), java does not release
memory to system (and it's a normal behaviour according to this
http://stackoverflow.com/questions/22912063/automatically-release-unused-memory-in-imagej-fiji).
Now when I load the same kind of image (same size), the JVM should use
the free memory released by the previously closed image isn't it ? I
have the feeling that it does not because, when I move the time slider
on the second image, it's very slow....

Some numbers:

- image size : 4.6GB
- Fiji memory after start : 0.3GB
- Fiji memory after loading first image: 6.9GiB
- Fiji memory after closing first image: 6.9GiB
- Fiji memory after loading second image: 6.9GiB

My hypothesis is that when loading an image, Fiji/SCIFIO look at free
memory in the system but not in the JVM. Is that possible ? Did you
already noticed that ?

PS: I have attached to this mail some properties of my system.

Best,

--
Hadrien Mary

Ph.D student in Biology
Tournier-Gachet Team
CNRS - LBCMCP - UMR 5088

Université de Toulouse - Bât. 4R3B1
118, route de Narbonne - 31062 Toulouse

_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel

Properties.txt (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: About memory usage with Fiji/JVM

Mark Hiner
Hi Hadrien,

>My hypothesis is that when loading an image, Fiji/SCIFIO look at free
>memory in the system but not in the JVM. Is that possible ?

Yes, this is almost certainly what's happening. Unfortunately I am not aware of a way to differentiate between memory that is still in use, and memory that just hasn't been deallocated.

By default in these situations SCIFIO will look at the available memory, see that there's not enough to open the dataset and open it virtually. Note that it actually would be possible to open the full dataset - as the memory allocation approached the JVM's limit, it would automatically be released.

Right now this can be worked around programmatically by specifying a PLANAR or ARRAY[1] ImgMode in the SCIFIOConfig[2]. The Bio-Formats importer lets you specify whether the image should be opened virtually or not. The only other option is to click the status bar to force garbage collection.

We also have plans to improve the SCIFIO importer[3] to expose these settings in the UI. Ultimately the hope is that virtual (cell) images will incur minimal performance penalties though[4], making this issue moot.

Thanks!
- Mark

On Mon, Jan 26, 2015 at 5:57 AM, Hadrien Mary <[hidden email]> wrote:
Hi,

New SCIFIO I/O system in Fiji/IJ2 automatically detects wether virtual
stack is needed according to system memory available and image size
(if I understood correctly).

Virtual stack is fine but it always better to work without when you
can. For example working with timelapse stacks, sliding time is much
faster when image is not a virtual stack (and it makes sense since RAM
memory access is much faster than hard drive memory access...).

Saying that I noticed something: when I close a big image (no virtual
stack because moving time slider is fast), java does not release
memory to system (and it's a normal behaviour according to this
http://stackoverflow.com/questions/22912063/automatically-release-unused-memory-in-imagej-fiji).
Now when I load the same kind of image (same size), the JVM should use
the free memory released by the previously closed image isn't it ? I
have the feeling that it does not because, when I move the time slider
on the second image, it's very slow....

Some numbers:

- image size : 4.6GB
- Fiji memory after start : 0.3GB
- Fiji memory after loading first image: 6.9GiB
- Fiji memory after closing first image: 6.9GiB
- Fiji memory after loading second image: 6.9GiB

My hypothesis is that when loading an image, Fiji/SCIFIO look at free
memory in the system but not in the JVM. Is that possible ? Did you
already noticed that ?

PS: I have attached to this mail some properties of my system.

Best,

--
Hadrien Mary

Ph.D student in Biology
Tournier-Gachet Team
CNRS - LBCMCP - UMR 5088

Université de Toulouse - Bât. 4R3B1
118, route de Narbonne - 31062 Toulouse

_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel



_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel
Reply | Threaded
Open this post in threaded view
|

Re: About memory usage with Fiji/JVM

Hadrien Mary
Thank you Mark for the answer.

About ImgMode I wonder what is the difference between these three modes :

- ArrayImgFactory
- CellImgFactory
- PlanarImgFactory

Sorry if it is something obvious but I often see these modes without
knowing how they work.

Best,

--
Hadrien Mary

Ph.D student in Biology
Tournier-Gachet Team
CNRS - LBCMCP - UMR 5088

Université de Toulouse - Bât. 4R3B1
118, route de Narbonne - 31062 Toulouse


On Fri, Mar 13, 2015 at 7:43 PM, Mark Hiner <[hidden email]> wrote:

> Hi Hadrien,
>
>>My hypothesis is that when loading an image, Fiji/SCIFIO look at free
>>memory in the system but not in the JVM. Is that possible ?
>
> Yes, this is almost certainly what's happening. Unfortunately I am not aware
> of a way to differentiate between memory that is still in use, and memory
> that just hasn't been deallocated.
>
> By default in these situations SCIFIO will look at the available memory, see
> that there's not enough to open the dataset and open it virtually. Note that
> it actually would be possible to open the full dataset - as the memory
> allocation approached the JVM's limit, it would automatically be released.
>
> Right now this can be worked around programmatically by specifying a PLANAR
> or ARRAY[1] ImgMode in the SCIFIOConfig[2]. The Bio-Formats importer lets
> you specify whether the image should be opened virtually or not. The only
> other option is to click the status bar to force garbage collection.
>
> We also have plans to improve the SCIFIO importer[3] to expose these
> settings in the UI. Ultimately the hope is that virtual (cell) images will
> incur minimal performance penalties though[4], making this issue moot.
>
> Thanks!
> - Mark
>
> [1]
> https://github.com/scifio/scifio/blob/scifio-0.21.1/src/main/java/io/scif/config/SCIFIOConfig.java#L124
> [2]
> https://github.com/scifio/scifio/blob/scifio-0.21.1/src/main/java/io/scif/config/SCIFIOConfig.java#L370-373
> [3] https://github.com/imagej/imagej-plugins-commands/issues/4
> [4] https://github.com/scifio/scifio/issues/201
>
>
> On Mon, Jan 26, 2015 at 5:57 AM, Hadrien Mary <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> New SCIFIO I/O system in Fiji/IJ2 automatically detects wether virtual
>> stack is needed according to system memory available and image size
>> (if I understood correctly).
>>
>> Virtual stack is fine but it always better to work without when you
>> can. For example working with timelapse stacks, sliding time is much
>> faster when image is not a virtual stack (and it makes sense since RAM
>> memory access is much faster than hard drive memory access...).
>>
>> Saying that I noticed something: when I close a big image (no virtual
>> stack because moving time slider is fast), java does not release
>> memory to system (and it's a normal behaviour according to this
>>
>> http://stackoverflow.com/questions/22912063/automatically-release-unused-memory-in-imagej-fiji).
>> Now when I load the same kind of image (same size), the JVM should use
>> the free memory released by the previously closed image isn't it ? I
>> have the feeling that it does not because, when I move the time slider
>> on the second image, it's very slow....
>>
>> Some numbers:
>>
>> - image size : 4.6GB
>> - Fiji memory after start : 0.3GB
>> - Fiji memory after loading first image: 6.9GiB
>> - Fiji memory after closing first image: 6.9GiB
>> - Fiji memory after loading second image: 6.9GiB
>>
>> My hypothesis is that when loading an image, Fiji/SCIFIO look at free
>> memory in the system but not in the JVM. Is that possible ? Did you
>> already noticed that ?
>>
>> PS: I have attached to this mail some properties of my system.
>>
>> Best,
>>
>> --
>> Hadrien Mary
>>
>> Ph.D student in Biology
>> Tournier-Gachet Team
>> CNRS - LBCMCP - UMR 5088
>>
>> Université de Toulouse - Bât. 4R3B1
>> 118, route de Narbonne - 31062 Toulouse
>>
>> _______________________________________________
>> ImageJ-devel mailing list
>> [hidden email]
>> http://imagej.net/mailman/listinfo/imagej-devel
>>
>

_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel
Reply | Threaded
Open this post in threaded view
|

Re: About memory usage with Fiji/JVM

Mark Hiner
>Sorry if it is something obvious but I often see these modes without knowing how they work.

No problem at all.

This option decides which underlying ImgLib2 data structure is preferred to back the image data: Array[1], Planar[2], or Cell[3].

Cell is the virtual option that reads planes on demand from disk (or cache). Array is faster but historically some operations could only operate on PlanarImgs.

You might find the ImgLib2 examples[4] helpful, as well.

On Fri, Mar 13, 2015 at 1:54 PM, Hadrien Mary <[hidden email]> wrote:
Thank you Mark for the answer.

About ImgMode I wonder what is the difference between these three modes :

- ArrayImgFactory
- CellImgFactory
- PlanarImgFactory

Sorry if it is something obvious but I often see these modes without
knowing how they work.

Best,

--
Hadrien Mary

Ph.D student in Biology
Tournier-Gachet Team
CNRS - LBCMCP - UMR 5088

Université de Toulouse - Bât. 4R3B1
118, route de Narbonne - 31062 Toulouse


On Fri, Mar 13, 2015 at 7:43 PM, Mark Hiner <[hidden email]> wrote:
> Hi Hadrien,
>
>>My hypothesis is that when loading an image, Fiji/SCIFIO look at free
>>memory in the system but not in the JVM. Is that possible ?
>
> Yes, this is almost certainly what's happening. Unfortunately I am not aware
> of a way to differentiate between memory that is still in use, and memory
> that just hasn't been deallocated.
>
> By default in these situations SCIFIO will look at the available memory, see
> that there's not enough to open the dataset and open it virtually. Note that
> it actually would be possible to open the full dataset - as the memory
> allocation approached the JVM's limit, it would automatically be released.
>
> Right now this can be worked around programmatically by specifying a PLANAR
> or ARRAY[1] ImgMode in the SCIFIOConfig[2]. The Bio-Formats importer lets
> you specify whether the image should be opened virtually or not. The only
> other option is to click the status bar to force garbage collection.
>
> We also have plans to improve the SCIFIO importer[3] to expose these
> settings in the UI. Ultimately the hope is that virtual (cell) images will
> incur minimal performance penalties though[4], making this issue moot.
>
> Thanks!
> - Mark
>
> [1]
> https://github.com/scifio/scifio/blob/scifio-0.21.1/src/main/java/io/scif/config/SCIFIOConfig.java#L124
> [2]
> https://github.com/scifio/scifio/blob/scifio-0.21.1/src/main/java/io/scif/config/SCIFIOConfig.java#L370-373
> [3] https://github.com/imagej/imagej-plugins-commands/issues/4
> [4] https://github.com/scifio/scifio/issues/201
>
>
> On Mon, Jan 26, 2015 at 5:57 AM, Hadrien Mary <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> New SCIFIO I/O system in Fiji/IJ2 automatically detects wether virtual
>> stack is needed according to system memory available and image size
>> (if I understood correctly).
>>
>> Virtual stack is fine but it always better to work without when you
>> can. For example working with timelapse stacks, sliding time is much
>> faster when image is not a virtual stack (and it makes sense since RAM
>> memory access is much faster than hard drive memory access...).
>>
>> Saying that I noticed something: when I close a big image (no virtual
>> stack because moving time slider is fast), java does not release
>> memory to system (and it's a normal behaviour according to this
>>
>> http://stackoverflow.com/questions/22912063/automatically-release-unused-memory-in-imagej-fiji).
>> Now when I load the same kind of image (same size), the JVM should use
>> the free memory released by the previously closed image isn't it ? I
>> have the feeling that it does not because, when I move the time slider
>> on the second image, it's very slow....
>>
>> Some numbers:
>>
>> - image size : 4.6GB
>> - Fiji memory after start : 0.3GB
>> - Fiji memory after loading first image: 6.9GiB
>> - Fiji memory after closing first image: 6.9GiB
>> - Fiji memory after loading second image: 6.9GiB
>>
>> My hypothesis is that when loading an image, Fiji/SCIFIO look at free
>> memory in the system but not in the JVM. Is that possible ? Did you
>> already noticed that ?
>>
>> PS: I have attached to this mail some properties of my system.
>>
>> Best,
>>
>> --
>> Hadrien Mary
>>
>> Ph.D student in Biology
>> Tournier-Gachet Team
>> CNRS - LBCMCP - UMR 5088
>>
>> Université de Toulouse - Bât. 4R3B1
>> 118, route de Narbonne - 31062 Toulouse
>>
>> _______________________________________________
>> ImageJ-devel mailing list
>> [hidden email]
>> http://imagej.net/mailman/listinfo/imagej-devel
>>
>


_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel