[JIRA] Dev – Where are OSWorkflow Transition Screens?

Found a piece that isn’t well documented. But as usual, I’m explaining my steps and then the solution. It’s no real big deal anyway. 🙂

Here’s the background:
Mission was to write a Groovy script that returns all the transition screens used across all workflows of a given JIRA workflow scheme (for documentation purposes). This looked pretty easier, until a collection of ‘com.opensymphony.workflow.loader.ActionDescriptor’ came up without any linked JavaDocs. That looked like a dead end.

Then, with a little bit of googling I came across ActionDescriptor source code, which wasn’t helpful either. getView() method looked promising because ‘Transition View’ is a synonym used in JIRA UI for ‘Transition Screen’. But disappointingly it was just returning string ‘fieldscreen’ for every transition. Aw.. snap! 😦

Next attempt is called the ‘Bruteforce method’ (not java.lang.reflect.Method of course 😉 ).

Since I was too lazy to read across the ActionDescriptor’s source code:

// workflow is type of com.atlassian.jira.workflow.JiraWorkflow
// I'm not explaining all my code here
workflow.getAllActions().each {
  it.class.getDeclaredMethods().grep {
    it.name.matches( "^get.*" )
  }.each {
    log.debug ( it.name )
  }
}
getAutoExecute
getConditionalResults
getPreFunctions
getUnconditionalResult
getMetaAttributes
getValidators
getPostFunctions
getRestriction
getView
getName

It didn’t took much of time. With a bit of guessing I made my first attempt with getter method ‘getMetaAttributes()’. And it had the eye-catching piece I was looking for!

workflow.getAllActions().each {
  log.debug ( it.getMetaAttributes().toString() )
}
[opsbar-sequence:10, jira.description:, jira.fieldscreen.id:10000]
[opsbar-sequence:20, jira.description:, jira.fieldscreen.id:20000]
[opsbar-sequence:40, jira.description:, jira.fieldscreen.id:30000]
[opsbar-sequence:60, jira.description:, jira.fieldscreen.id:30000]

getMetaAttributes() returns a HashMap. So, finally…

def screens = [:]
workflow.getAllActions().each {
  def screenId = it.getMetaAttributes()["jira.fieldscreen.id"]
  if ( screenId != "" ) {
    def workflowAction = workflow.name + " - " + it.name
    def workflowActions = []
    if ( screens[screenId] ) workflowActions = screens[screenId]
    workflowActions.add ( workflowAction )
    screens[screenId] = workflowActions
  }
}
for ( e in screens ) {
  log.debug ( "Transition Screen: " + e.key )
  e.value.each {
    log.debug ( "  Workflow action: " + it )
  }
}

Nice! This HashMap (‘screens’) is all what I wanted. 🙂 Each transition screen, and then where it has been used.

So one last thing… I also wanted to have a one last peek into ActionDescriptor’s source code on GitHub. Only the writeXML() method gave an indication that metaAttributes is actually a HashMap. There was absolutely no indication that screen association information is kept as a ‘meta attribute‘.

Bottom line(s):

  1. If you are a JIRA developer and whenever you want to find screens associated to a workflow, don’t use getView(). Use getMetaAttributes() and then look for the attribute with key “jira.fieldscreen.id”. In other words, transition screen is stored as a meta attribute in each transition. Screen is referred by its numeric ID as a string.
  2. Since OpenSymphony doesn’t seem to go away from JIRA, and some already claim that OpenSymphony is dead, it would be worthwhile if Atlassian maintains their own repository of OSWorkflow docs and special remarks.
Advertisements

JIRA project migration with Google Spreadsheets

JIRA to JIRA project migration — most of the time this has been a headache when it comes to customer negotiations and compromises.

In my experience, usually it starts with a discussion round to identify what customer exactly needs. Some customers expect everything, including issue change history, comments, votes, etc. In this case we have to use the XML project import method. With status, resolution, priorities and custom field mapping this is a headache. In the near-perfect JIRA project migrations I’ve performed, I remember I used Groovy scripts to merge field values, convert additional issue types to labels, and etcetera. If you choose XML import, it needs a lot of technical work and beforehand testing when you have a demanding customer and a JIRA instance that isn’t expected to change much.

The other type of customers are simple-living minimalist people. 😉 They keep things simple, needs are simple and they just need the contents to be brought over. They are okay with basic things such as issue summary, description, type and status.

The method I’m going to discuss today is for the customers who lie in-between the two types of customers above. In my particular case I met a customer who needs comments and one of their custom fields to be brought over. File attachments and issue change history were not necessary for them. In this case downloading JIRA issues from Issue Navigator into Excel and using it as a CSV source won’t do the perfect import.

This led me to develop a custom Google Spreadsheets script. It uses JIRA REST API and fetches project issues into the spreadsheet. Then, it can be downloaded as CSV and imported into JIRA. Here’s a reduced version of the code I wrote – just to demonstrate:

var jql = encodeURIComponent("project = ABC ORDER BY key ASC");
var rows = [];
var s = 0;
while (true) {
  var response = fetchJIRA (baseurl, "/rest/api/2/search?jql=" + jql + "&maxResults=200&fields=*all%2Ccomment&startAt=" + s);
  // fetchJIRA () is a custom function I wrote 
  // to call JIRA REST API and return JSON.
  // It is not explained in this article.
  var max = response.maxResults;
  var c = response.issues.length;
  var t = response.total;
  if (c == 0) break;

  response.issues.forEach (function (i) {
    var row = [];
    row.push (i.key);
    row.push (i.fields.summary);
    // .. Include other fields
    rows.push (row);
    s = response.startAt + c;
  });
}
rows.unshift (["Key", "Summary", ...]);

var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
var sheet = spreadsheet.insertSheet("Issues");
var iR = sheet.getRange(1, 1, rows.length, rows[0].length);
iR.setValues(rows);

Actually it does more!

  • Status/ issue type/ resolution/ user account/ select-list custom field value mapping:
    Using a function like this maps statuses on-the-fly fetching, so we don’t have to worry about creating a workflow.

    function statusMapper (param) {
    var mapping = {
      "Open":"Backlog",
      "Done":"Closed",
      "Resolved":"Closed",
      "Validate":"In Verification",
      "Planned":"Defined"
    };
    var ret = mapping[param];
    if (ret == undefined) ret = param;
    return ret;
    }
    row.push (statusMapper (i.fields.status.name));
    row.push (typeMapper (i.fields.issuetype.name));
    row.push (i.assignee ? userMapper (i.assignee.name) : "shaakunthala");
    // ^^ Handy if target project doesn't allow unassigned issues.
    // It maps user accounts on-the-fly, 
    // while assigning all unassigned issue to a default user account.
  • Linking parent and sub-task on-the-fly. The script will do what’s asked in the Atlassian’s official documentation. After fetching issues, ParentIDs will be added using VLOOKUP spreadsheet formula applied by the script.
    var issueid_col = sheet.getLastColumn();
    var newCol = issueid_col + 1;
    var parentkey_cell1 = sheet.getRange(2, issueid_col - 1, 1, 1).getA1Notation();
    var formula ="=IFERROR(VLOOKUP(" + parentkey_cell1 + "," + issuesRange.getA1Notation().replace (/[0-9]/g, "") + "," + issueid_col + ",FALSE), \"\")";
    sheet.getRange(1, newCol, 1, 1).setValue("Parent ID");
    sheet.getRange(2, newCol, 1, 1).setFormula(formula)
         .copyTo(sheet.getRange(3, newCol, sheet.getLastRow() - 2, 1));
  • Mapping additional custom fields to labels. This is useful when you want to avoid creating any custom fields.
    if (i.fields.customfield_30000 != null)
      labels_array.push (i.fields.customfield_30000.value);
  • Comments, Affects Versions, Fix Versions, Labels, etc. can be fetched into different sheets of the same spreadsheet to be imported separately. The reason for this is to avoid large and much complicated CSV files that aren’t much human-readable.
    • In this case, still, target project doesn’t have to be empty. If target project is different, relationship between multiple CSV imports can be handled with issue key inserted to a text custom field as a reference ID.

So, here’s the pros and cons.

Pros:

  1. What’s being transferred over are in human-readable format in the spreadsheet. It may be useful reference. This gives you a better way to verify any mapping and additional labeling you have done.
  2. Google Spreadsheets will take care of the CSV syntax and escaping. You don’t have to worry a thing about CSV errors. Just code what needs to sit in each column.
  3. Less time consuming compared to XML import, but brings over more data compared to minimal CSV import.

Cons:

  1. Limitations of Google Spreadsheets: Google will terminate the script if it runs for more than 6 minutes. This can happen with larger projects. In this case you can use a Python script instead. But you’ll need to write additional code to map parents and sub-tasks.

Reference: 

Official documentation by Atlassian: https://confluence.atlassian.com/adminjiraserver071/importing-data-from-csv-802592885.html

Notes:

There could be third-party add-ons that can perform hassle-free project migrations. If you can purchase such an add-on, this kind of approaches may not be necessary.

There is a method to import JSON instead of CSV. This syntax appears to be different from the issue object returned by the REST API. Therefore you’ll still need a custom script to perform JSON import.

[JIRA] Lexorank integrity issues?

So, this is loosely another part of the unofficial series ‘fixing a corrupted JIRA db‘.

However, this is a different case. This JIRA instance (7.1.x) was created by importing a huge XML backup (> 1 million issues). Due to an unknown reason, ActiveObjects didn’t import. That means, JIRA Software (aka JIRA Agile) no longer functioned normal.

It was soon discovered that Lexorank integrity checks fail. For the record, following integrity checks failed:

Marker rows present in table for rank field
Marker rows correctness check
Marker rows in valid bucket check
Issue rows in valid bucket check
Balance status check

Now what? I made an unsuccessful attempt of uninstalling JIRA Software, then dropping all database tables/ triggers/ sequences with the prefix AO_60DB71_ and then re-installing JIRA Software. This didn’t help.

Then, I wanted to study the under-the-hood stuff of Lexorank management. I already knew of two database tables involved, AO_60DB71_LEXORANK and AO_60DB71_LEXORANKBALANCER.

First I created a custom field of type ‘Global Rank’ and named it ‘Rank2’.

Then, I navigated to JIRA Administration –> System –> Lexorank management. Now I could see two Rank fields under the Balancing section. I balanced them once again and executed the integrity checks. The old one still failed the above checks, but the new one passed all checks!

Now, let’s take a look at the database. Following query returned two rows, that looked like an upper marker (1|zzzzzz:) and a lower marker (1|000000:). Both were associated with the field ID of the newly created ‘Rank2’ custom field.

select * from AO_60DB71_LEXORANK;

What I did was a very simple thing. I updated the field ID of the two records so they are now associated to the old Rank field.

update AO_60DB71_LEXORANK
set FIELD_ID = (select ID from CUSTOMFIELD where CFNAME = 'Rank');

commit;

Finally, to clean up I deleted the newly created Rank2 custom field. Simple!

After a JIRA restart Lexorank started to function normal, with all the integrity checks passed. I hope my approach will help someone who has run into the same problem.

[JIRA] Broken Permission Schemes? (better way)

Yesterday I blogged about how to fix JIRA Permission Schemes of a corrupted JIRA database. The observation was from atlassian-jira.log, but in my case I soon found out the problem was much greater in my case.

All projects lost their workflow scheme, issue type screen scheme, and field configuration scheme, and project category associations. This led me to peek into the database directly. The NODEASSOCIATION table had only 20 – 30 rows – those are the permission schemes I fixed yesterday. Also, all issues lost their components, affected versions and fix versions.

So how do I fix this? I can fix issue type screen scheme association without a hassle, notification schemes without a hassle, but workflow schemes? Even automating the restoration may run into dead ends given the nature of steps of workflow scheme association through UI. The only option seemed to be restore the corresponding records to NODEASSOCIATION table from the most recent database backup.

This is actually safer than meddling with other database tables in JIRA, because NODEASSOCIATION has no primary key, so no chance of primary key violation errors after starting up JIRA. With all sounded good, here’s my steps.

  1. From the most recent backup of the same JIRA instance, export the NODEASSOCIATION table. With Oracle SQL Developer there’s an option to export records in ‘INSERT’ format, which generates a SQL script.
    • If you use another database/ client that can’t export into ‘INSERT’ format, you can export to a CSV or spreadsheet and then use spreadsheet formulae to generate INSERT queries.
    • In my case I had to exclude where SINK_NODE_ENTITY = ‘PermissionScheme’ because I’ve already fixed that.
  2. In the target JIRA database, run the above SQL script and commit. Easy! 😀
  3. Restart JIRA and re-index.

This added some 60,000+ records with all entity association you’ll find out with this query:

select distinct source_node_entity, sink_node_entity, association_type from nodeassociation order by 1 asc;

That’s it! Not 100%, but this worked up to my expectation. The key reason for this being hassle-free and easy is, NODEASSOCIATION table has no primary key. This is actually much much better than using Groovy, because you have to pay for the add-on. 🙂

[JIRA] Broken Permission Schemes?

Edit (24 May): Spoiler alert! There’s a better way.

I came across a corrupted JIRA database with all projects lost their permission schemes.

From the database, I believe this happens when corresponding records were deleted from the NODEASSOCIATION table. Whatever the cause is, you’ll notice the following log entry in atlassian-jira.log when this happens:

2017-05-23 02:57:18,363 WARN [permission.DefaultPermissionSchemeManager]: No permission scheme is associated with project ''

The problem is, (in JIRA 7.1.x) when you navigate to Project Administration –> Permissions to associate it with a scheme, you’ll see an empty page. For a newly created project Default Permission Scheme should show up, but now it absolutely has no permission scheme. It’s lost.

So, here’s the expensive but easy solution. Use the Script Runner add-on. Execute this Groovy code in the script console and it will fix everything in a minute!

import com.atlassian.jira.component.ComponentAccessor;
import com.atlassian.jira.ComponentManager;
import com.atlassian.jira.bc.project.ProjectService;
import com.atlassian.jira.bc.project.ProjectService.UpdateProjectSchemesValidationResult;

def projectService = ComponentManager.getComponent(ProjectService.class);
def currentUser = ComponentAccessor.getJiraAuthenticationContext().getLoggedInUser();

Long permissionSchemeId = 10000; // ID of the permission scheme that you wish to associate to each project
Long notificationSchemeId = null;
Long issueSecuritySchemeId = null;
ProjectService.UpdateProjectSchemesValidationResult validationResult = projectService.validateUpdateProjectSchemes(currentUser, permissionSchemeId, notificationSchemeId, issueSecuritySchemeId);

ComponentAccessor.getProjectManager().getProjectObjects().each {
  projectService.updateProjectSchemes(validationResult, it);
}

As it can be understood from the code, you can fix Notification Schemes and Issue Security Schemes using the same way.

[JIRA] Delete projects from XML backup using Python

[Experimental]

If you are a JIRA system administrator, have you ever come across the requirement of partially exporting JIRA into XML – probably for importing individual projects into another JIRA instance?

If yes, this might work for you. However, my work is incomplete and experimental. This worked with a small JIRA instance (100 issues), and failed with a huge JIRA instance (1 million issues).

Here’s the Python script I wrote for deleting projects from JIRA XML backup:

#!/usr/bin/python

from lxml import etree

#xp = etree.XMLParser(encoding='utf-8', recover=True)
#doc = etree.parse('entities.xml', xp)
doc = etree.parse ('entities.xml')

# Array of project keys
plist = ['ABC', 'DEF', 'GHI']

for x in plist:
    for p in doc.xpath ("//Project[@key!=\'" + x + "\']"):
        #p.get entities and delete
        pi = p.get("id")
        for i in doc.xpath ("//Issue[@project=\'" + pi + "\']"):
            ii = i.get("id")
            for cfv in doc.xpath ("//CustomFieldValue[@issue=\'" + ii + "\']"):
                cfv.getparent().remove(cfv)
            for cg in doc.xpath ("//ChangeGroup[@issue=\'" + ii + "\']"):
                cgi = cg.get("id")
                for ci in doc.xpath ("//ChangeItem[@group=\'" + cgi + "\']"):
                    ci.getparent().remove(ci)
                cg.getparent().remove(cg)
            for fa in doc.xpath ("//FileAttachment[@issue=\'" + ii + "\']"):
                fa.getparent().remove(fa)
            for il in doc.xpath ("//IssueLink[@source=\'" + ii + "\']"):
                il.getparent().remove(il)
            for il in doc.xpath ("//IssueLink[@destination=\'" + ii + "\']"):
                il.getparent().remove(il)
            for na in doc.xpath ("//NodeAssociation[@sourceNodeEntity=\'Issue\' and @sourceNodeId=\'" + ii + "\']"):
                na.getparent().remove(na)
            for wl in doc.xpath ("//Worklog[@issue=\'" + ii + "\']"):
                wl.getparent().remove(wl)
            for c in doc.xpath ("//Action[@issue=\'" + ii + "\']"):
                c.getparent().remove()
            i.getparent().remove(i)
        for na in doc.xpath ("//NodeAssociation[@sourceNodeEntity=\'Project\' and @sourceNodeId=\'" + pi + "\']"):
            na.getparent().remove(na)
        for c in doc.xpath ("//Component[@project=\'" + pi + "\']"):
            c.getparent().remove(c)
        for v in doc.xpath ("//Version[@project=\'" + pi + "\']"):
            v.getparent().remove(v)
        for a in doc.xpath ("//ProjectRoleActor[@pid=\'" + pi + "\']"):
            a.getparent().remove(a)
        for pk in doc.xpath ("ProjectKey[@projectId=\'" + pi + "\']"):
            pk.getparent().remove(pk)
        p.getparent().remove(p)

f = open ('entities.new.xml', 'w')
f.write (etree.tostring(doc, pretty_print=True, xml_declaration=True))
f.close()

Before you run this, you need to unzip the JIRA XML backup which is a zip archive containing two or three XML files. Put the script file in the same directory where you have the unzipped XML files and run it. It will produce output into file ‘entities.new.xml’. Once completed, replace ‘entities.xml’ by ‘entities.new.xml’, remove the Python script and re-pack the folder into a new zip file. Now it is ready to be imported into another JIRA instance.

Please note that this is incomplete and I cannot guarantee 100% success. You can easily follow the structure of this code and improve it by yourself.

[JIRA] Five guidelines for managing custom fields

Here’s a little post for JIRA Administrators.

One of the major culprits, if JIRA runs slow, is the number of custom fields. What makes your JIRA to have so many custom fields? Because JIRA is customizable and so easy doing that, whenever a colleague of yours calls and say we want a new custom field, you go ahead and you create them.

Your organization grows, number of projects grows, number of people grows, and you continue with this practise, and eventually, because you didn’t follow a properly defined process, JIRA soon becomes unmanageable.

So, if you plan to start rolling out JIRA within your organization, here are some guidelines for custom fields:

  1. Define a process. First. Must.Before you roll out JIRA within your organization, despite how small it is, you need to have a process to manage JIRA.
    No valid business justification – no custom field. If majority of projects/ people is not benefited – no custom field.
  2. Not more than a couple of JIRA administrators.If you are in a small organization just one or two people having administration access to JIRA is enough. More people having this privilege will lead to misuse, and a common victim of this misuse is custom fields.
  3. Create a project within JIRA to manage JIRA.Sounds silly but your colleagues can request custom fields through it rather than verbally or through email. This project can be used not only to track custom fields but also many other changes within JIRA.
  4. Have custom fields documented.If you have Confluence, it is the ideal place for this. Just maintain a simple document (wherever you can) which will contain a list of custom fields in tabular format. Include business justification, important parameters, etc. Keep it consistent.
    It’s easy and simple rather than going through JIRA’s administration console for any reference, and your boss can (should be able to) understand it too! (Transparency is important)
  5. Periodic audits and tagging.Periodic audits help re-determining the purpose of each existing custom field. Let’s assume that you have a custom field called ‘X’ created for some of the projects, and some time later these projects die. As a result the valid business justification for X’s existence dies. But you can’t delete X because defunct projects still use it. Now, what will happen, by any chance you accidentally use X for a new project? You will never see your way to delete it.
    So, if a custom field is no longer required, rename it, and prepend its name with a tag like ‘[Archived]’. If you need a lengthy description and/or you don’t want users to notice the tag, you can instead append a HTML comment (<!– yada yada –>) inside the field’s description.
    Have a SQL query prepared for auditing custom fields, and with your creativity it should be able to produce a nice report out of tags.

What if you don’t follow the above? Once users have started using a custom field you can’t delete it without losing already entered data. Your human users are not happy of losing something they previously had. Which means, you eventually end up with a huge set of custom fields. At this point, rather than cleaning up JIRA by deleting unnecessary custom fields, you may have to work on performance engineering JIRA, or perhaps spend money on high performance hardware.

As a Sri Lankan proverb says, cut it off (a seedling) with your nailtip today, or tomorrow you’re gonna need an axe.

Anything else I missed, please comment below.