Author Archives: Marcel Austenfeld

Transfer data from R to Python with PyRserve and Bio7

29.08.2018

Recently I discovered the package PyRserve for Python which connects Python with R using Rserve.

This is extremly useful because Bio7 already integrates Rserve and has special GUI interfaces available to transfer, e.g.,  data from spreadsheets, ImageJ image and selection data (also georeferenced),  Java simulation data, etc.

With this new Rserve connection this data can now easily transferred from Java and R to a Python workflow.

Python as a language is already supported in Bio7 with an integration of Jython (which runs on the Java platform a supported JVM language like Groovy, JavaScript, Scala, etc. and has a direct access to the Bio7 API).
In additon native Python can be executed with the default Bio7 Python editor if configured in the native preferences of Bio7 (running as an external process in a Bio7 Python shell if enabled).

However since Bio7 is based on Eclipse a more powerful editor for Python/Jython can be installed with PyDev which is a complete Python IDE for Eclipse and available as an Eclipse plugin which can be installed with the Bio7 (Eclipse) Update Manager

Here a video how PyDev can be installed in Bio7:

Please activate JavaScript to view this video.
Video-Link: https://youtu.be/-kySAN9doXQ

Update Site PyDev: http://www.pydev.org/updates/

If PyDev is installed you can create Python projects in Bio7 and execute Python and Jython scripts depending on the settings in it’s own process (e.g. with the action editor context menu-> Run As->Python Run) .

In addition it is possible to use a special Bio7 toolbar action (visible if the PyDev editor or Bio7 Python editor is opened) to interpret the content of the PyDev editor in the current Bio7 Jython or Python process (which is of no importance for the following use of PyRserve!).

PyRserve can be installed easily using the pip command in the python shell:

pip install PyRserve

or by using PyDev with the preferences dialog (see screenshot below).

After the installation we can start RServe from within R as usual ,e.g., with:

library(Rserve)
run.Rserve()

To connect Python to Rserve the following commands are suffcient:

import pyRserve
conn = pyRserve.connect()

More examples can be found in the PyRserve documentation.

However since the Bio7 functionality can be easily extended with scripts (which can be added dynamically to a menu copying them to a special scripts location, see screenshot below) I decided to write two short Groovy scripts to start and stop Rserve not controlled by the Bio7 Java connection interface (and if necessary disconnect R from a current Bio7 controlled Java connection).

The ‘Start_PyRserve’ script starts R and Rserve if not alive in the Bio7 R console and also disconnects from a running Java Rserve connection if activated.

Note that in Bio7 only one connection to Rserve is allowed (Rserve cooperative mode).

The ‘Stop_PyRserve’ script reconnects Python (closed in the Python scripts!) to shutdown Rserve and returns to the Bio7 console and R (Pydev has it’s own console).

This shutdown command is excecuted by an external Python process. To get the path for the Python interpreter we also have to set the Bio7 Python path (not PyDev!) in the Bio7 preferences (see Windows screenshot below – normally not necessary on Linux or when the Python interpreter is on the OS PATH!).

With this two scripts it is now (more secure) possible to start Rserve and use the Bio7 Java interface, e.g., to transfer image data to R (ImageJ to R), start the Python connection (which disconnects Java) and use the image data inside a Python workflow (see the second example).

Here a first example script to transfer random numbers (in PyDev use the context menu action ‘Runs As->Python Run’ of the editor to execute the Python script) with a PyRserve connection.

import pyRserve

conn = pyRserve.connect()
result = conn.eval("result<-runif(100)")
conn.close()
print (result)

Here another example script to plot a transferred image matrix in Python with matplotlib (8-bit, grayscale as double matrix) if available in the R workspace: (see this old video how to do this with Bio7):

import pyRserve 
import matplotlib.pylab as plt

conn = pyRserve.connect()
matrix = conn.eval("tryCatch({matrix<-imageMatrix}, error = function(e) {return ('NULL')})")
conn.close()
if matrix != 'NULL':
    # we have to transpose the image for a correct view!
    matrix = matrix.T
    plt.imshow(matrix)
    plt.colorbar()
    plt.show()

Here a summary of the installation and configuration steps necessary for Bio7 to use PyRserve and execute Python scripts with the powerful PyDev editor:

  1. Install the PyDev editor and configure the Python path
  2. Install PyRserve (pip install PyRserve)
  3. Configure the Python path of Bio7 (Preferences Bio7->Preferences Native)
  4. Install the Groovy Start/Stop scripts for PyRserve (simply unzip Examples_PyRserve.zip in one scripts location – Open ‘Scripts->General->Open_Script_Location’ from the main Bio7 menu or in the context menu of the R-Shell view open General->Open_Script_Location’. Put them in an extra folder for a submenu!
  5. Start the ‘Start_PyRserve’ script to start Rserve (not Bio7 controlled!)
  6. Execute a Python script which connects and disconnects PyRserve
  7. Stop the server and shutdown Rserve with the ‘Stop_PyRserve’ script

Download Start/Stop PyRserve scripts here

 

 

 

Bio7 2.9 Released

16.08.2018

A new release of Bio7 is available. The new Bio7 2.9 release comes with a plethora of new R features and bugfixes.

Screenshot MacOSX with dark theme and opened ‘R ImageJ Analysis’ perspective

Screenshot ‘R’ perspective Windows and opened R-Shell code completion

Release Notes:

General:

  • Based on Eclipse 4.8
  • Improved the dark theme and the layout of the dark theme in many places
  • All editor font colors are now changed automatically to default optimized colors when using the dark theme
  • New ‘Switch Theme’ action available to easily switch to the dark theme or back to the default theme
  • Java updated to 1.8.181
  • New Fullscreen/Hide Menu action available (Main menu “Window” – Key: Strg+Shift+ 4)
  • New Hide and Show main menu actions available (useful to hide added plugin menus again)
  • Added pdf reader option “Okular” to open a R pdf plot with the okular application (used when the SWT browser + open external preference is selected)
  • Added a new Table API method to transfer a matrix array form Java more efficiently
  • Improved the WorldWind search function (using the photon geocoding API)
  • Improved the default font size for Swing components on Retina displays (e.g., ImageJ components)

R:

  • Updated R for Windows to version 3.5.1
  • Added a new plot option to plot huge plots in a temporary folder and open them in ImageJ virtually (disk resident). The folder will automatically be opened after plotting (plot images have to be deleted manually!)

ImageJ

  • ImageJ Updated to 1.52f18
  • ImageJ toolbar improved for the dark theme (see screenshots)
  • Added a new ImageJ detach image menu in the ‘Window’ menu of the ImageJ-Canvas view

R Markdown:

  • Added syntax coloring of R markdown snippets
  • Added new context menu actions in menu “Text” (Find and replace, Toggle Block Selection, To Upper Case, To Lower Case)
  • Added a new context menu action to toggle word wrap, etc.

R editor:

  • Added a new warning and quickfix for possibly wrong comparisons of NULL, NA, NaN

  • Added a new option in the preferences for the new warning
  • Added new context menu actions in menu “Text” (Find and replace, Toggle Block Selection, To Upper Case, To Lower Case)
  • Improved the display for quickfix suggestion, warning and errors when the font is resized dynamically (especially for MacOSX)
  • The help browser for code completion popups now display help sites in dark mode if the Bio7 dark theme is enabled.

ImageJ Macro Editor

  • Added a general rename method to rename selected word occurences
  • Added new context menu actions in menu “Text” (Find and replace, Toggle Block Selection, To Upper Case, To Lower Case)
  • Added latest macro code completion function definitions

Download and Installation:

Windows:

Just download the *.zip distribution file from https://bio7.org and unzip it in your preferred location. Bio7 comes bundled with a Java Runtime Environment, R and Rserve distribution and works out of the box.

Linux:

Download and extract the installation file from https://bio7.org.
For Linux you have to install R and Rserve (see Rserve installation below!).

MacOSX:

Download and extract the installation file from https://bio7.org.

If you start Bio7 a warning or error can occur because of the changes how Apple treats signatures! To allow Bio7 to start see this instructions for Yosemite and Sierra:

OS X Yosemite: Open an app from an unidentified developer

macOS Sierra: Open an app from an unidentified developer

If you have still problems with Sierra see this solution!

In addition for MacOSX you have to install R and Rserve (see below!).

Linux and MacOSX Rserve (compiled for cooperative mode) installation:

To install Rserve open the R shell and then execute the menu action “Options -> Install Rserve (coop. mode) for R …” for different R versions. This will download an install Rserve in your default R library location, see video below (please make sure that your default Linux R library install location has writing permissions!). In cooperative mode only one connection at a time is allowed (which we want for this Desktop appl.) and all subsequent connections share the same namespace (default on Windows)!

Play Video

This video will be embedded from Youtube. The privacy policies of google apply.

Installation of Useful R Packages

R packages which are useful in combination with Bio7 can easily be installed with the R-Shell context menu “Packages” action:

R-Shell context menu->Packages->Install_Default_R_Packages

Bio7 Documentation

For more information about Bio7 please consult the soon updated Bio7 User Guide.

A plethora of Bio7 videotutorials for an introduction can be found on YouTube.

Debug Java in Bio7

27.06.2018

In Bio7 R and Java code can be easily combined in the Bio7 platform. For instance to create Graphical User Interfaces in Java (with SWT or JavaFX), transfer image pixel/selection data from ImageJ and finally doing the analysis parts in R code/scripts which can be called from within Java (an easy to use API on top of Rserve in Bio7 is available, see, e.g., Groovy examples here).

However sometimes it is necessary to debug the Java language as part of such combined work. In Bio7 (based on Eclipse RCP) the very powerful Java debugging tools from Eclipse are available by default.

In addition in Bio7 it is possible to debug dynamically compiled Java classes in the same Bio7 process (using Java like a scripting language is a special feature of Bio7).

The video at the bottom of this page demonstrates the use of a remote debugging connection on the same computer (localhost connection) to debug dynamically compiled/executed Java code.

Debugging a Java process which has already started (Bio7 in this case) is only possible if Bio7 is started beforehand with the following Java arguments inside a shell to start a debug server connection:

Windows (started from the current directory = Bio7 installation directory):
Bio7.exe -vmargs -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

Mac (given the full path to the Bio7 executable):
/Applications/Bio7.app/Contents/MacOS/Bio7 -vmargs -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

Linux (started from the current directory = Bio7 installation directory):
./Bio7 -vmargs -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

Furthermore the compiler debug information option has to be enabled in the Bio7 Java preferences (for the dynamic compilation process) and a seperate debug connection has to be started for this Java class (with the available Eclipse debug configuration dialog).

In the video below I (hopefully) demonstrate how you can (graphically) debug a simple cellular automata simulation like the Game of Life within a running Bio7 (Java) process.

Please activate JavaScript to view this video.
Video-Link: https://www.youtube.com/watch?v=vKAGBGuQKlI

Bio7 2.8 Released

30.05.2018

A new and improved release of Bio7 is available. The new Bio7 2.8 release comes with a plethora of new R features and bugfixes.

Release Notes:

General:

  • Updated Eclipse RCP to 4.7.3
  • Windows R update to version 3.5.0
  • ImageJ plugin updated to version 1.52d7
  • Added the ImageJ Edit perspective to the default opened perspectives
  • The R perspective is now selected as the opened perspective after startup of Bio7
  • Added an option to save all editors on close (default enabled on MacOSX)
  • Improved the main ‘Scripts’ menu to allow nested custom scripts in folders and subfolders which form new submenus
  • Improved the Table and R-Shell context scripts menu
  • Added some example scripts to the new scripts menu
  • Added an script to open the default script location in the OS file explorer
  • Simplified the different script locations (reduced to four locations in the Bio7 preferences)
  • Spatial import, export scripts moved to the main ‘Scripts’ and R-Shell script context menu (Deleted import, export script locations)

R:

  • Simplified the Bio7 R-Shell interface
  • Added a new view for some R templates
  • Code completion from loaded R packages can now be imported from the R-Shell or the R editor (both completions will be updated!)
  • Improved the reload code completion by only considering R functions (cleaner seperation of functions and datasets)
  • The updated R-Shell context menu can now dynamically be extended with custom scripts (updated dynamically). Folder and subfolders form new submenus.
  • Improved, bugfixed and extended the ‚Evaluate Line‘ action which was renamed to ‚Evaluate Line or Selection‘  action which now interprets selected R code, too.
  • Improved MacOSX ImageJ interaction (editor hoover focus)
  • Improved the call of the default plot device (quartz) in a native R connection on MacOSX (now default opened when, e.g., plotting, debugging, etc.)
  • Added a new menu to install Rserve for R > 3.5.0
  • New shutdown option to execute R code at shutdown (e.g., to save the current R workspace – a reload with the already existing startup option is possible, too!)
  • Improved the general error messages of the R-Shell view and the R editor evaluation. Now much simpler and printed cleaner!
  • Added an option to print the error stream of the Bio7 console to the R-Shell view (to the right output panel to get an output if the console is hidden!)
  • Fixed a bug with the shiny execution (removed annoying dialog)
  • Fixed many other bugs (unnecessary error messages, execution errors, MacOSX bugs, etc.)
  • R-Shell code completion improved (see editor code completion improvements below)
  • Removed the use of temporary files of the R-Shell and R editor code completion
  • Added an preference option to add arguments to the source command
  • Multiple selected R-Shell variables are now available in the R workspace (‘.r_shell_vars’ – a character vector) for the use in custom R scripts (e.g., to extend the R-Shell script context menu!)

R Editor:

  • Added a detection/quickfix function to install missing packages when a package load definition is in the editor file but the package is not installed

  • Added a quickfix to add a library definition when an unknown function is called (not defined and loaded but package with function is installed)
  • Added support for the the magrittr piping operator in code completion to autocomplete a given dataframe in the workspace (column completion)

  • Added code completion support for dataframes columns and rows, named list elements and named arrays (of arbitrary dimension)

  • Improved code completion to display current workspace variables
  • Classes of R objects are now displayed in the code completion context dialog
  • Editor variables and functions are now emphazised in code completion, too
  • Added a directory dialog to easily create a directory string for the editor file
  • Created new submenus for the organization of the context menu
  • Improved the hoover dialog (improved offset)
  • Improved the completion context of R objects (current workspace objects)
  • The styler package can now be used for an improved code formatting (requires UTF-8 Bio7 workspace on Windows – see preferences!)
  • Improved code completion for S3 and S4 objects (correctly display current cursor completion)

ImageJ:

  • Updated ImageJ plugin to version 1.52d7
  • Updated ImageJ2 libraries
  • Added new ImageJ macro functions to the code completion of the macro editor
  • Added the ImageJ Edit perspective to the default openend perspectives
  • Improved the editor hoover activation on MacOSX
  • Improved the ImageJ MacOSX touchpad resizing of the ImageJ panel

Java:

  • Updated JRE to 1.8.71
  • Improved the creation of Java projects and project imports (no annoying dialog anymore)
  • Improved the ‘Add Selected Libraries’ preference to add multiple Java libraries for dynamic compilation (e.g., to compile more embedded Eclipse library functions)
  • Now ImageJ2 libs have to be added to the dynamic compilation libraries for compilation (removed as default libraries)
  • Added a key shortcut (STRG + ‚+‘) to increase the SceneBuilder panel (2x)

Download and Installation:

Windows:

Just download the *.zip distribution file from https://bio7.org and unzip it in your preferred location. Bio7 comes bundled with a Java Runtime Environment, R and Rserve distribution and works out of the box.

Linux:

Download and extract the installation file from https://bio7.org.
For Linux you have to install R and Rserve (see Rserve installation below!).

MacOSX:

Download and extract the installation file from https://bio7.org.

If you start Bio7 a warning or error can occur because of the changes how Apple treats signatures! To allow Bio7 to start see this instructions for Yosemite and Sierra:

OS X Yosemite: Open an app from an unidentified developer

macOS Sierra: Open an app from an unidentified developer

If you have still problems with Sierra see this solution!

In addition for MacOSX you have to install R and Rserve (see below!).

Linux and MacOSX Rserve (compiled for cooperative mode) installation:

To install Rserve open the R shell and then execute the menu action “Options -> Install Rserve (coop. mode) for R …” for different R versions. This will download an install Rserve in your default R library location, see video below (please make sure that your default Linux R library install location has writing permissions!). In cooperative mode only one connection at a time is allowed (which we want for this Desktop appl.) and all subsequent connections share the same namespace (default on Windows)!

Please activate JavaScript to view this video.
Video-Link: https://youtu.be/tF7HbRBRIFQ

Installation of Useful R Packages

R packages which are useful in combination with Bio7 can easily be installed with the R-Shell context menu “Packages” action:

R-Shell context menu->Packages->Install_Default_R_Packages

Bio7 Documentation

For more information about Bio7 please consult the soon updated Bio7 User Guide.

A plethora of Bio7 videotutorials can be found on YouTube.

 

R Improvements for Bio7 2.8

18.05.2018

The next release of Bio7 adds a lot of new R features and improvements. One minor change is that the default perspective after the startup of Bio7 now is the R perspective to emphazise the importance of R within this software.

The R-Shell view has been simplified and the R templates have been moved in it’s own simple view for an improved usability (see screenshot from R perspective below).

In addition the context menu has been enhanced to allow the creation of submenus from scripts found in folders and subfolders (recursively added) which you can create for a menu structure.
Scripts can be added created in R, JavaScript, Groovy, Jython, BeanShell, ImageJ Macros.
Java (with dependant classes) can be dynamically compiled and executed like a script, too.

Several improvements have also been added to the R-Shell and the R editor for an easier generation of valid R code. The R-Shell and the R editor now display R workspace objects with it’s class and structure in the code completion dialog (marked with a new workspace icon – see below).

R-Shell:

R editor:

In the R editor a new quick fix function has been added to detect and install missing packages (from scanned default packages folder of an R installation – has to be enabled in the Bio7 R code analysis preferences).

Also the detection of missing package imports are fixable (when a function is called but the installed package declaration is missing in the code but the package is installed to deliver the function).

The code assistance in the R-Shell and in the R editor now offers completions for, e.g., dataframes (columns) in the %>% operator of piped function calls:

In addition code assistance is available for list, vectors, dataframes and arrays of named rows and columns, etc., when available in the current R environment.

Code completion for package functions can now easily added with the R-Shell or the R editor which loads the package function help for both interfaces. The editor will automatically be updated (see updated editor marking unknown functions in screencast below).

Numerous other features, improvements and bugfixes have been added, too.

Bio7 2.8 will hopefully be available soon at:

https://bio7.org

Overview videos on YouTube