Last updated: Feb 19, 2021
This page will be updated with questions (and answers) as they arise. Please check back regularly. If you do not find an answer to your question here, you are welcome to contact the IJCAI-21 program chair at firstname.lastname@example.org, but please make sure that you have read IJCAI-21 CFP and the current FAQ first.
[Many parts are inherited from IJCAI-20 FAQ]
Yes. If the paper was on arXiv at least one month before the abstract submission deadline of IJCAI-21, and it is not cited by the submitted paper.
There are many reasons for a submission not appear on arXiv immediately; however, once you
submit to arXiv, you will have a “submission date” (e.g., https://arxiv.org/abs/1702.08835, you’ll
see in the upper left that the submission date is noted as “28 Feb, 2017”). IJCAI-21 only
requires that your “submission date” (not the “arXiv publication date”) is at least one month
before the abstract submission deadline of IJCAI-21.
The submission of your last update must be at least one month before the abstract submission deadline of IJCAI-21.
Yes. This applies more generally to papers presented in any non-archival venue (that is, as long as there are no formally published proceedings with a publisher).
IJCAI requires that all submissions be anonymized. To facilitate this, we require that the authors follow both the letter and spirit of the normal anonymization guidelines. Any paper that obviously violates this requirement will be rejected without review.
Yes. Resubmissions from at any peer-reviewed conference within the past 6 months have to follow the “Resubmissions” terms in the call for papers, otherwise the paper may be rejected without further review.
Only the reviews/response of the last conference is required.
Please make sure you have read the “Resubmission” part of CFP to know the details about the resubmission.
To submit resubmission file PDF, first upload your submission. You will then be able to upload resubmission file from the author console as a supplementary file. There are two entries for supplementary files: one is “TechnicalAppendix”, which is the normal supplementary file for the main paper; and the other one is “ResubmissionFile”, to which you should upload the resubmission file. Note that the maximal size of the resubmission PDF is 10M.
No file can be submitted after the full-paper submission deadline.
Your resubmission information will be visible to reviewers only during the discussion phase after they submit their initial reviews.
A paper is considered a ‘student paper’, and thus a potential winner of the ‘Best Student Paper Award’, when the first author is a student who, at the time of submission, has not obtained any PhD degree yet. When submitting a paper, this should be indicated by marking the appropriate checkbox.
Yes. If, for instance, the short paper is a two-column two-page paper. If the ‘short’ paper is longer than that, but no longer than half of the IJCAI paper, then the extension should be significant enough. Significance will be assessed by reviewers. In any case, the short version has to be cited and it must be clear that the IJCAI submission is an extension.
For anonymity of the IJCAI-21 submission, in the citation please hide the authors’ names, and hide the place where the abstract got published. However, you need to mention that this is an extension of a published abstract, and submit the published abstract as supplementary file (with authors’ names/affiliations and publication venue removed). For example, you should mention:
This submission is an extension of a published extended 2-page abstract [AAA].
Ref-AAA: XXX. Short paper title. YYY, 2020. [see: supplementary file]
Title: It is highly recommended not to go above two lines in the IJCAI format. Three lines is a strict maximum.
Abstract: There is no word limit for abstracts. However, 200 words is a nice upper limit that authors should always apply.
By ‘abstract submission’ we mean that you should register your paper by giving a title and a short abstract. This abstract is intended for helping program committee members getting the papers they are competent for reviewing. There are no abstract-only submissions, nor any paper-only submissions: in all cases, you must submit an abstract, and then you must submit a full paper. (See abstract submission and paper submission deadlines on the web site).
You can always include references in the six main body pages, in addition to the last reference page.
For reference, \small is OK, but no smaller than \small, otherwise, the submission may be rejected without review.
Please ask your co-authors to log into the submission system and fill in author information. Once your co-authors log in, they can see all papers under their names (if they have multiple accounts, please check all accounts). If your co-authors found any submission which had not got their agreement for submission, they should report to program chair immediately
GoogleScholar page, Research Gate page, or other webpages containing your publications. Or if you do not have any publications before, the webpage of your research lab where your name appears can be used.
PCs (Program Committee members): participate in only full-paper review phase: write reviews and participate in discussions.
SPCs (Senior Program Committee members): participate in both summary-reject and full-paper review phases. In summary-reject phase: fill in checkbox form and (optional) write brief reviews; in full-paper review phase: write reviews and participate discussions.
ACs (Area Chairs): participate both summary-reject and full-paper review phase. In summary-reject phase: fill in checkbox form and (optional) write brief reviews; in full-paper review phase: Coordinate reviewers, write meta-reviews and make recommendations.
SACs (Senior Area Chairs): A very small group of leading experts who help cross-check the reviews, meta-reviews and recommendations of ACs.
(A)PCC: Program Chair and Associate Program Chairs: IJCAI has a single Program Chair governing the whole program of the conference. 3 Associate Program Chairs with different expertise are appointed to help the Program Chair to cover the whole AI field.
Nominate or self-nominate PCs are welcome. The nominee should have published at least three papers in premier AI conferences such as IJCAI, AAAI, NeurIPS, ICML, CVPR, ICCV, ACL, KDD, etc. in the past five years. A PhD degree in AI fields is expected. Senior PhD students who have published at least five papers in premier AI conferences can also be nominated as candidates. Nomination/self-nomination should go to email@example.com Approved PCs should register to the CMT system with invitation sent from Workflow Chair, and fill in all required information immediately. Later, some senior PCs will be upgraded to SPCs.
Many factors will be considered, such as how long the candidate has been working in AI fields, how rich the experience of the candidate has for publishing in premier AI conferences, how rich the experience of the candidate has for serving AI conferences. In addition, an important factor is how many papers are submitted to IJCAI-21; we plan to have a reasonable workload for SPCs in the summary reject phase, and thus, some PCs may be selected to SPCs after the abstract submission deadline. Another important factor is in what AI subarea the candidate is working; we plan to have a balanced workload across different subareas. Thus, it is important to fill in CMT information as early as possible for PCs who wish to be selected as SPCs.
IJCAI-21 review process will be triple blind at the level of Area Chairs (AC), Senior Program Committee members (SPC), and Program Committee members (PC), i.e., AC/SPC/PC cannot see the identities of authors and other AC/SPC/PCs, and vice versa. Author identities are also invisible to Senior Area Chairs (SACs) and vice versa.
During the discussion phase, AC/SPC/PCs should avoid disclosing your own identity. You can use CMT number to call other reviewers such as Meta-Reviewer1, Reviewer2, etc. To compensate for information of the hidden reviewer names, you can see the information of “how many years have you worked in Artificial Intelligence?” of other reviewers handling the same submission.
Google scholar ID: Enter your own Google Scholar page, for example, https://scholar.google.com/citations?user=rSVIHasAAAAJ&hl=zh-en, then Google Scholar ID can be extracted from the url between “=” and “&”, namely, “rSVIHasAAAAJ”.
Semantic Scholar ID: Enter your own Semantic Scholar page, like: https://www.semanticscholar.org/author/Z.-Zhou/145624000, then Semantic Scholar ID can be extracted from the end of the url, namely, “145624000”.
To find your DBLP ID, a detailed instruction is given at the CMT official website: https://cmt3.research.microsoft.com/docs/help/general/external-profile.html.
The rapid growth of IJCAI and many other AI conferences has overwhelmed the review resource. Thus, a summary reject procedure will be executed for IJCAI-21. In this process, each submission will be assigned to a group of senior experts (ACs and SPCs) to judge whether the submission has a chance to be accepted by IJCAI-21. The group-size will be decided based on the number of submissions and the size of senior program committee. An SAC will cross-check the group judgment, finally governed by (A)PCC. Submissions passing this procedure will go into the full-paper review phase.
This summary reject process will try to help authors save time for revision and resubmission earlier to other venues. IJCAI-21 will provide checkbox information and encourage reviewers to write brief reviews. Note that by submitting a paper, authors admit to be aware of the possibility of receiving a summary rejection notification without any written review.
You are senior people so, if you are from classic AI areas, with a long tradition of publication at IJCAI, you know that people submit their best papers. For areas which have grown a lot in the last few years (NLP, vision, machine learning), it is less evident. A rule of thumb to decide if a paper has the IJCAI standard of quality is to ask yourself whether the paper is good enough for ACL/ICCV/ICML. If not, it is not good enough for IJCAI either.
All papers violating the rules specified in the CFP must be rejected.
Any paper that has clear weaknesses that we do not expect from IJCAI papers should be
summary rejected. Examples of weaknesses can be:
– too badly written/organized
– not motivated
– not positioned wrt related approaches
– too incremental
– not evaluated (depending on the contribution, it could be experimental, theoretical, etc)
– no technical contribution, just an application/tuning of existing techniques (except if the
problem it solves has a clearly demonstrated high impact on society)
– clearly not belonging to the area declared by the author(s)
All reviewers for the summary reject procedure are senior experts (SPCs and ACs), and you will have about 10 days to read the papers and make your judgment. Moreover, in this process, each submission will be assigned to a group of senior experts (maybe 10; the group-size will be decided based on the number of submissions and the size of SPC/AC) to judge whether the submission has a chance to be accepted by IJCAI-21; an SAC will cross-check the group judgment, finally governed by (A)PCC. Thus, the decision to summary reject the paper is not in your own hands only, and please feel free to express your judgement via votes and checkbox and (optional) brief comments.
As a reviewer (SPC/AC) for the summary reject procedure, you are requested to vote whether the paper will be summary rejected or go to full-paper review procedure, and select items for checkboxes. Moreover, you are encouraged to write some brief comments (optional). All these information will be visible to authors of summary rejected submissions.
All papers violating the rules specified in the CFP should have been detected during the summary reject phase. In case you still find one (e.g., the identity of the authors is made explicit in the paper or supplementary files), then please quickly send me an email so that we can confirm the rejection.
The assignment algorithm considers your bids, subject areas matching, as well as reviewer workload balancing, etc. It tries to optimize the global assignment. If you did not bid, or entered positive bids for few papers, or if you entered bids for which many other people entered bids as
well, then the system was not able to find enough papers for you based on your bids. Some subject areas are quite broad so that you may be assigned a paper that is in your subject area but not in your exact area of expertise. Bear in mind that reviewing for a conference as large scope as IJCAI, you cannot have only papers of people working on exactly the same stuff as you. We trust your expertise to have an opinion on the quality of your papers.
Warning: If you see that a paper does not fit at all its primary subject area, please recommend rejection. We want to get rid of these papers where authors selected a bad subject area just in the hope they will get reviewers who do not have the skills to see the weaknesses of their paper.
If you still think you cannot review a paper because it is too far from your expertise, please send me an email as soon as possible. I’ll assign it to someone else. Obviously, the longer you wait, the more difficult the reassignment. (The same thing applies if you discover a conflict of interest, or if you cannot review a paper for any other reason.)
In principle, NO. For a big conference like IJCAI, it is very difficult to control the COI if subreviewers are allowed before we have more resource for the organization. In any case, it is yourself who will be responsible for the review.
Most likely it means that the paper was withdrawn by its authors.
A good review should contain a small paragraph summarizing the contribution, so that the AC and the SAC in charge of this paper can quickly see what it is about. Then, a second short paragraph summarizing the reasons for your overall score. Finally, the body of your review, that is, the detailed explanation of what is good/bad in this paper and why.
IJCAI-21 is a conference that bears on all aspects and subareas of Artificial Intelligence. AI is broad and it is sometimes difficult to evaluate whether a particular paper is within the scope of IJCAI-21 or not. In case of doubt, use your best guess and report an explanatory comment in your review. In any case, if you are assigned a paper that you think is not relevant enough to AI, please don’t decide to stop reviewing it without consulting me beforehand.
The purpose of the response phase is not to enter in a discussion with the authors. When reviewing the submission, you should judge the submission according to the quality of the initial submission only, instead of the potential revision (we cannot guarantee that any revision will reach the expectation definitely). So please make your review focusing on the primary contributions and qualities of the submission, and do not regard the response phase as a communication/discussion with authors. As stated in the CFP page, author responses will be limited to clear factual errors in the reviews, and are not intended to create a dialogue between reviewers and authors. Author responses are visible to Area chairs and SAC.
Only for very very crucial issue which seriously influence your judgment and you want a clarification, you can use the response phase to obtain this information. You must explicitly ask for this information with the format described below:
[***Response phase: Author please clarify what data were used to tune the parameters in the
first experiment ***]
The area chair, who will receive the response from the authors, will send it to you. Please use this possibility with parsimony because the main purpose of the response phase is to give the opportunity to authors to warn the area chair about factual errors or non-ethical reviews, not to initiate a discussion.
It is difficult to give a precise answer, as this depends a lot on the area. As a general rule, epsilon improvement papers are not the best fit for a conference paper. A small improvement on performance due to a small improvement of an algorithm is usually not enough (except if the application problem has really strong impact on science/society). A rule of thumb:
– new technique, small improvement: generally ok.
– small adaptation of a known technique, big improvement: generally ok.
– small adaptation of a known technique, small improvement: generally not ok.
Application papers are welcome at IJCAI but IJCAI is the place to publish research papers, not engineering papers.
If the use of this AI technique is novel on this kind of application problem and the problem solved in this application is formalized and abstracted enough so that other applications could benefit from the same technique, it is research. If the paper doesn’t completely satisfy the previous criterion but it clearly explains why the used technique works on this problem, again it can be considered as research. In those cases, acceptance depends on how much the application is important, how much the improvement is significant, and to which extend it can be generalized to other problems.
If the paper doesn’t satisfy the previous criteria but the AI technique used significantly improves the solving of this application and this application has major impact on industry/society, we can consider it is enough for publication to IJCAI.
Otherwise, we would say reject.
Certainly not. Some important results have simple proofs. Some results have long and involved proofs but are of little interest.
The existence of non-anonymous preprints (on arXiv, social media, websites, etc.) will not result in rejection provided that the preprints (and its last update) existed at least one month before the abstract submission deadline of IJCAI-21. Note that the submission to CMT must always be anonymized regardless of whether a preprint has been released. Reviewers will be instructed not to actively look for such preprints, but encountering them will not constitute a conflict of interest. Preprints (or updates) uploaded no more than one month before the IJCAI-21 abstract submission deadline will be considered as violating anonymity.
No, if the short paper is not longer than a two-column two-page paper. If the ‘short’ paper is longer than that, but no longer than half of the IJCAI paper, then the extension should be significant enough. In any case, the short version has to be cited (in a way which does not violate anonymity) and it must be clear that the IJCAI submission is an extension.
Authors of papers that have been rejected at any peer-reviewed conference within the past 6 months must declare the resubmission by including a cover letter. The cover letter should summarize the main reasons for rejection and should describe the changes the authors have made to address the reviewers’ comments. The cover letter should be put in a separate file, along with the previous reviews and previous anonymized rejected submission. Such resubmissions must contain, in a single PDF file (different from the IJCAI-21 paper file):
– the cover letter, starting with the title of the rejected paper, the tracking number (if any), and
– the name of the conference to which this paper was submitted,
– the full reviews from the rejected conference,
– the anonymized version of the rejected paper.
This resubmission information will be visible only during the discussion phase after you submit initial review.
Note that the maximal size of this resubmission PDF is 10M.
The cover letter should give you an idea of what was wrong for acceptance in the previous submission, and what changes the authors have made to improve their paper (or why they have not made them, e.g. disagreement with reviewers opinion).
As a reviewer of IJCAI, you are allowed to reject a paper because the authors have not made the effort to improve the rejected submission. However, you are not obliged to accept the paper for the single reason that the authors have addressed the comments of the rejected submission. IJCAI is an independent conference with its own standards of quality. If you think the paper is still too weak, you can reject it independently of what the cover letter says.
If you find a paper rejected from any peer-reviewed conference within the past 6 months but does not declare the resubmission, the paper should be rejected and please send me an email.
Please continue to review it. We don’t regard this situation as a COI. Reviewer for previous version of the submission is at a nice place to judge whether the current submission has been improved dramatically or it is simply a rolling of dice gambling the conference review system. Note that a submission which has been rejected at any peer-reviewed conference within the past 6 months must declare the resubmission and attach the previous review as well as an explanation about how authors have improved the work. The resubmission file will be visible to you during discussion period.
Also see: FAQ.[Program Committee]
There are two phases for the paper review: summary reject phase and full paper review phases.
In the summary reject phase, AC will act like a reviewer: to vote, input to checkbox, and write
brief comments (optional). See FAQ.[summary reject].
In the full-paper review phase: Each paper will be reviewed by a group of reviewers (SPC and
PC), and ACs are not supposed to review the paper yourself. The main responsibility of ACs is
to coordinate review and discussion, write a meta-review with a recommendation (cross
checked by a SAC); only when there were a missing review you failed to chase, a review from
you as replacement is needed.
Yes. ACs have been assigned a few papers for which they have to oversee the review process. They will check the quality of reviews, they will handle the authors’ response, they will monitor the discussion between reviewers, and they will write a meta-review with a recommendation to SAC.
Ideally, a good review should contain a small paragraph summarizing the contribution, so that the SAC and the AC in charge of this paper can quickly see what it is about. Then, a second short paragraph summarizing the reasons for the overall score. Finally, the body of the review, that is, the detailed explanation of what is good/bad in this paper and why.
We see two reasons to ask the reviewers to modify their review:
The first reason is if the review contains rude or impolite sentences. Even if a reviewer doesn’t like a paper, the phrasing should in any case remain polite and respectful of the work that the authors have done.
The second reason is when a review is non-informative. We don’t want to see a review that simply summarizes the paper or gives a non-supported opinion (e.g., “very good paper, the best paper in my set”). You should insist to reviewer that you will not take into account such reviews for the decision.
Here are how overall scores appear to reviewers in the full-paper review phase in CMT:
6. Strong Accept (I don’t see how it could be rejected)
5. Accept (good paper. I can argue for accepting it)
4. Weak Accept (marginally above the acceptance threshold. Rejecting it would not be that bad)
3. Weak Reject (marginally below the acceptance threshold. Accepting it would not be that bad)
2. Reject (Not good enough. I can argue for rejecting it)
1. Strong Reject (such a paper would really hurt the reputation of IJCAI)
All such papers should have been already discovered, but in case it still happens, please send me an email.
AI is broad and it is sometimes difficult to evaluate whether a particular paper is within the scope of IJCAI-21 or not. In case a reviewer thinks a paper is not relevant enough to AI, please discuss with SAC in charge of the paper.
The existence of non-anonymous preprints (on arXiv, social media, websites, etc.) will not result in rejection provided that the preprints (and its last update) existed at least one month before the abstract submission deadline of IJCAI-21. Note that the submission to CMT must always be anonymized regardless of whether a preprint has been released. Reviewers will be instructed not to actively look for such preprints, but encountering them will not constitute a conflict of interest. Preprints (and updates) uploaded no more than one month before the IJCAI-21 abstract submission deadline will be considered as violating anonymity.
You have to check whether the response really points out a factual error in a review or unethical issues. If unethical issues, we strongly suggest that you contact the SAC in charge of the paper and see what has to be done. In the case of factual error, you should assess the importance of the error. Is it the reason used by the reviewer to propose rejection? If yes, you have to ask the reviewer to fix the problem with their review. In all other cases, please write in your meta-review that you have read the response but that it the raised concern had no influence on the decision.
In principle, No. The response phase is an opportunity given to authors to defend themselves against unethical reviews or factual errors used to reject the paper. For this reason, the response will be hidden to reviewers. Only the SAC and the AC in charge of the paper can see the response.
Only for very very crucial issue which seriously influence reviewer’s judgment, reviewer can use the response phase to obtain this information. For this, reviewer must explicitly ask for this information with the format described below:
[***Response phase: Author please clarify what data were used to tune the parameters in the
first experiment ***]
If your reviewer has made such a request, and if you receive the response from the authors about that, please copy the text and send it to reviewer. We have clearly emphasize to reviewers that they should use this possibility with parsimony because the main purpose of the response phase is to give the opportunity to authors to warn the area chair about factual errors or non-ethical reviews, not to initiate a discussion. If you find a review raised too many requests, please feel free to request reviewer to focus on the most crucial issue that requires a clarification.
First, read all the reviews, as well as the response (if any). In case of ‘non-informative’ reviews, please refer to FAQ.When should I ask a reviewer to modify their review?.
If at least three reviews are informative enough and the opinions in these reviews converge to something other than “borderline paper”, this will not be difficult: propose a recommendation to the reviewers, and if none objects, it will be final.
If there is no agreement about the paper, or if you think there is not enough information to make a recommendation, then you should initiate the discussion by summarizing the disagreement points and ask all reviewers to discuss.
It is important that all reviewers who took part to the evaluation of the paper participate in the discussion, especially those who have an opinion different from the main trend. If someone is unresponsive, you should encourage them to say something.
Authors were given the opportunity to respond to the AC (See FAQ What should I do if the authors have sent a response?).
If unethical review, you have to evaluate the level of the problem. You may feel that it is true that the review contains comments that are not appropriate (e.g., very short and rude review without elaborated arguments, or: “I reject this paper because not compared to [ABC et al., to appear], or: “I reject because there exists an arXiv of this paper”) and then decide to weaken the importance of that review, or ask the reviewer to revise their review. You may otherwise feel it is a delicate issue (e.g., the authors claim the reviewer has a COI with this paper). In this case please inform the SAC.
If the response raises an erroneous statement, you have to use your own judgement on whether you will reveal this to the reviewers or not. In many cases the solution is to raise the point among reviewers as if it was coming from you (e.g., “Reviewer #3 says that this algorithm is not complete, but I see a proof of correctness in page 3. Can you clarify?”). If Reviewer #3 does not refute or does not recognize s/he was wrong, you can ask the help of the other reviewers to elucidate the issue.
Yes, it is fine. However, it should only happen if the other opinions have convinced the reviewer that her/is review was not reflecting the real value of the paper. It is important that reviewers DO NOT change their review and/or score just to adapt to the majority, or to adapt to the decision.
NO. For several reasons. First, it is better for the author to see that the decision for their paper has not emerged from a unanimity of positive or a unanimity of negative opinions. Second, in case of not easy decision, the SAC will have a look at the reviews and discussion. If all reviews finally say the same thing, the SAC cannot understand what was the disagreement.
You are the one deciding when the discussion is over. There are two possible states when a discussion closes:
a. When you propose a decision and that everyone agrees (or, at least nobody objects), then the discussion can be considered to be closed and decisive.
b. In some rare cases, the discussion will remain inconclusive, with nothing seeming to be able to change that. Then you can decide to close the discussion as well. You will have to send a detailed report to the SAC.
Discussions should be ended and your recommendations must be available by April 7. For difficult cases, ask the help of the SAC in charge of the paper as soon as possible.
There is no formal way of closing the discussion. You simply close the discussion by posting a message such as:
“The decisions seems clear now. I will propose to accept/reject this paper.”
“After the discussion it seems that we all agree that the paper is borderline.”
“‘We should probably stop discussing now, as it seems unlikely we can reach an agreement. I’ll label the paper ‘Undecided’ and I’ll interact with the SAC in charge of the paper.”
Here are the possible labels:
“Accept”, “Borderline”, “Reject”, “Undecided”.
‘Accept’ and ‘Reject’ mean that the reviews and discussion allowed you to have a clear understanding of the pros and cons, and that you have resolved possible factual errors pointed out by authors that could affect final decision. If the decision is not obvious from the reviews (that is, the discussion has provided additional information you used for deciding), please write a meta review. The decision will be considered final, unless the SAC wants to discuss it.
‘Borderline’ means that the discussion has converged to this: the paper could perhaps be included in the proceedings, but the decision may depend on global preferences on the set of accepted papers (number of papers already accepted, relative acceptance rate across subfields etc.). In this case, please summarize the pros and cons in the confidential comments box. You may also send a private email to the SAC and discuss with them about the paper. BEWARE: ‘BORDERLINE’ DOES NOT MEAN THAT THERE IS NO AGREEMENT ON THE PAPER. It means that there IS an agreement that the paper is borderline.
‘Undecided’ means that on April 7, the discussion came to an end, but with no agreement and you are unable to make a decision.
If the decision was straightforward and the reviews contain all the information used to decide, do not write a meta review.
In all other cases, you should write a meta review. The meta review should summarize the issues raised during discussion, and in the case of ‘Accept’ and ‘Reject’ (only in these cases) you must also provide the reason why you made such a decision.